Sequential learning, advanced

Configure policies, rewards, delayed credit, exploration, training, and frozen evaluation.

Configure a Domain for actions that change later states and receive delayed outcomes. Begin with the starting declaration, run the query–execute–feedback loop, then adjust only the settings your task requires.

Sequential learning is Adapt-1’s episode-aware policy path for state-changing actions and delayed outcomes. This advanced guide explains how to configure and operate it through the Domain API. When Adapt-1 forms the supported state-dependent action-value and delayed-credit structure from evidence, the Discovery documentation refers to that capability as Sequential Discovery.

The objective is not to find one universal configuration. A good learner is a correctly scoped state-action-reward contract, followed by settings that match the task’s horizon, feedback density, nonstationarity, risk tolerance, and data budget.

Sequential learning does not require the whole Domain ontology to be discovered. The public action space, approved state paths, reward semantics, and episode boundaries can be authored, discovered where supported, or mixed. The acquisition schedule is separate again: sequential state can form inside a zero-start run, form in a separate trajectory acquisition phase, or continue from a separately acquired checkpoint.

1. When to use sequential learning

Enable sequential learning when all of the following are true:

• the system repeatedly chooses among two or more policies or actions;
• actions influence later observations;
• action quality depends on the current state;
• reward may be delayed, sparse, or distributed over an episode;
• events can be grouped into episodes and ordered by step;
• every action can eventually be linked to a measured outcome.

Examples include control, trading, game play, operations scheduling, treatment sequencing, multi-step remediation, and adaptive workflows.

Use ordinary contextual policy learning instead when each decision is independent and receives an immediate outcome. Use structured transition learning when the main output should be a prediction of the next value rather than a selected policy. Use both when the application needs next-state forecasts and policy selection, but keep their output contracts distinct.

Sequential learning does not register a separate sequential entry in learning_state.active_subsystems. It is an episode-aware model path inside feedback_policy. A Domain can therefore report feedback_policy and structured_transition as its active subsystems while sequential learning is enabled. When the sequential candidate trains and passes validation, inspect learning_state.subsystems.feedback_policy.model.report.model_type for sequential_q_mlp.

Choose zero-start learning, separate acquisition, or both

Setup	Use it when	Main constraint
One zero-start live task	Online adaptation inside that task is the goal, and feedback arrives early enough to affect later choices	One long episode is still one episode and cannot supply episode-held-out validation
Zero-start repeated episodes	Safe exploration and continuing improvement are part of the declared run	Keep one learner scope and use a new episode ID at each real reset; later episodes still use state earned inside the same zero-start run
Separate trajectory acquisition, then freeze	The first held-out action should use learned sequential value, or test-time exploration is unsafe or unavailable	Split acquisition and evaluation episodes before ingestion, preserve action coverage, and stop writes during evaluation
Separate acquisition checkpoint plus live episodes	Prior trajectories form state outside the new run, while the deployed process can drift	Preserve a fixed starting checkpoint so current-run adaptation can be measured separately

The complete declared run determines the acquisition label. A later episode is not warm-started merely because earlier episodes in the same zero-start run produced retained state.

Under the current defaults, the sequential candidate needs at least eight distinct episodes and reserves at least two episodes for validation. A Domain can declare a higher minimum_episodes. A single test episode may update ordinary contextual evidence when policy admission succeeds, but it does not establish validated sequential_q_mlp behavior.

A separate trajectory acquisition phase helps only when the state fields, actions, reward, terminal meaning, and useful action-value relationship transfer into the new run. Use a zero-start task-local scope when each independently redrawn task has a different hidden mapping and the current context does not identify that difference.

Historical trajectories use the same event and feedback contract as live trajectories. Replay approved records into fresh Domain state, preserve the true episode and step order, wait for asynchronous fitting, and inspect the model report before freezing. See Choose a learning setup for the schedule and scope decision.

2. The learning contract

Each sequential transition must provide this logical tuple:

(episode_id, step, current_context, relation, chosen_policy,
 next_context, step_reward, terminal)

The normal online loop uses two calls:

1. POST /domains/{domain_id}/query supplies the current context and returns a selected policy plus a decision_id.
2. POST /domains/{domain_id}/feedback returns the observed transition and reward.

A decision_id gives the strongest sealed binding to the earlier query, but policy admission can recover decision-time context through other supported routes. Current runtime context sources are a valid decision_id (decision), a valid target_memory_id (memory), or explicit structured context (request). Sequential readiness is downstream of this policy-admission step.

The default field mapping is:

Logical value	Default path in feedback
Episode ID	metadata.episode_id
Ordered step	metadata.step
Successor state	values.next_state
Per-step reward	values.step_reward
Terminal flag	values.terminal

All five values must be present and correctly typed for a record to be eligible for sequential training:

• episode_id: string or other stable identifier;
• step: finite number, normally a zero-based integer;
• next_state: JSON object;
• step_reward: finite number;
• terminal: JSON Boolean, not the strings "true" or "false".

Scope rule

Keep the same authenticated owner and domain_id when learning should carry across episodes. Use metadata.episode_id to separate environment episodes.

On the hosted API, the bearer token determines the effective session; changing the body session_id does not create a new tenant or learner. Use a new Domain for an independent task-state history, or a separate credential for a separate identity. The examples use session_id: ignored for compatibility.

3. Recommended starting declaration

The following declaration is a general starting point for a delayed-reward policy task. Replace the state fields, policies, policy features, and reward limits with values from the real application.

{
  "domain_id": "sequential-controller-v1",
  "session_id": "ignored",
  "description": "Select a control policy from state and delayed outcomes.",
  "schema": {
    "entities": ["controller", "workload"],
    "relations": ["controls"],
    "signals": ["queue_depth", "error_rate", "capacity", "step_reward"],
    "event_types": ["state", "feedback"]
  },
  "hypotheses": [
    {
      "name": "conservative controller",
      "relation": "controls",
      "policy": "conservative",
      "policy_features": {
        "intensity": 0.25,
        "risk_class": "low"
      },
      "when": [],
      "predicts": ["stable service with limited intervention"],
      "falsified_by": [],
      "weight": 1.0
    },
    {
      "name": "balanced controller",
      "relation": "controls",
      "policy": "balanced",
      "policy_features": {
        "intensity": 0.55,
        "risk_class": "medium"
      },
      "when": [],
      "predicts": ["balanced recovery and intervention cost"],
      "falsified_by": [],
      "weight": 1.0
    },
    {
      "name": "aggressive controller",
      "relation": "controls",
      "policy": "aggressive",
      "policy_features": {
        "intensity": 0.9,
        "risk_class": "high"
      },
      "when": [],
      "predicts": ["rapid recovery with higher intervention cost"],
      "falsified_by": [],
      "weight": 1.0
    }
  ],
  "learning": {
    "enabled": true,
    "context": {
      "feature_paths": [
        "values.queue_depth",
        "values.error_rate",
        "values.capacity",
        "values.operating_mode"
      ],
      "event_types": ["state"],
      "max_samples": 4096
    },
    "reward": {
      "aggregation": "weighted_mean",
      "components": [
        {
          "field": "values.step_reward",
          "goal": "maximize",
          "min": 0.0,
          "max": 1.0,
          "weight": 1.0,
          "required": true
        }
      ]
    },
    "policy": {
      "decay_half_life_seconds": 0.0,
      "transfer_strength": 0.0,
      "action_transfer_strength": 0.1,
      "min_context_observations": 4,
      "model_max_weight": 0.45,
      "model_ood_threshold": 0.75,
      "ood_confidence_penalty": 0.5,
      "exploration_mode": "auto",
      "exploration_strength": 1.0,
      "minimum_expected_reward": 0.0,
      "minimum_confidence": 0.0,
      "require_credible_dominance": false,
      "dominance_margin": 0.0,
      "abstain_on_ood": false
    },
    "credit_assignment": {
      "mode": "eligibility_trace",
      "discount": 0.95,
      "neutral_reward": 0.5,
      "minimum_weight": 0.01,
      "delay_path": "values.delay_steps"
    },
    "sequential": {
      "enabled": true,
      "episode_path": "metadata.episode_id",
      "step_path": "metadata.step",
      "next_context_path": "values.next_state",
      "reward_path": "values.step_reward",
      "terminal_path": "values.terminal",
      "discount": 0.95,
      "n_step": 6,
      "minimum_episodes": 10,
      "minimum_validation_skill": 0.1,
      "target_sync_interval": 10,
      "conservative_weight": 0.05
    },
    "latent_belief": {
      "enabled": true,
      "max_regimes": 8,
      "novelty_threshold": 0.55,
      "temperature": 0.25,
      "change_point_threshold": 0.55,
      "change_point_patience": 3,
      "historical_regime_weight": 0.05
    },
    "training": {
      "enabled": true,
      "min_samples": 64,
      "retrain_interval": 16,
      "dimensions": 96,
      "epochs": 100,
      "learning_rate": 0.04
    }
  }
}

Create it with:

export ADAPT1_BASE_URL="https://rei-neuroadapt-api.reilabs.org/api/v1"
export ADAPT1_API_KEY="YOUR_API_KEY"

curl -sS -X POST "$ADAPT1_BASE_URL/domains" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @domain.json

Configure the learner

4. Define the policy space correctly

Each selectable hypothesis needs:

• a name for humans and traces;
• the same relation for policies competing in one decision;
• a unique, non-null policy identifier;
• optional policy_features describing the action itself;
• optional when, predicts, and falsified_by clauses for interpretable conditions.

policy is the action identity. policy_features are observable action attributes that let the learner generalize between related actions. They must describe the candidate action, not its result.

Good policy features:

{
  "dose": 0.4,
  "duration_seconds": 30,
  "mode": "pulsed"
}

Poor policy features:

{
  "will_succeed": true,
  "future_reward": 0.9
}

Do not make the learned outcome or target action itself an autonomous structure target. A policy-less induced hypothesis is predictive evidence, not an executable action. Action selection must resolve to a declared policy.

5. Choose context features

learning.context.feature_paths determines which state fields are visible to contextual and sequential model training and inference.

Include fields that are:

• available before the decision;
• causally or predictively relevant to action value;
• represented consistently at query and feedback time;
• stable in name, unit, and type.

Exclude fields that are:

• generated after the action;
• direct encodings of the answer or reward;
• identifiers with no transferable meaning;
• timestamps when elapsed time or phase would be more useful;
• high-cardinality provenance that encourages memorization.

An empty feature_paths list uses all structured leaf fields. This is convenient for exploration but risky in production because IDs, logging fields, and accidental outcome leakage may enter the model. Explicit paths are recommended.

context.event_types filters stored events when a query asks Adapt-1 to infer the latest context. It does not replace explicit query context. Supplying context directly is the clearest and least ambiguous path.

context.max_samples bounds retained policy examples. For sequential tasks, size it to retain several representative episodes:

max_samples >= retained_episodes x representative_steps_per_episode

For 200-step episodes and 10 useful episodes, start at 2048 or 4096, not the default 512. Replay eviction balances episode groups and preserves failure evidence, but it cannot recover transitions that were never retained.

6. Define a real reward

Policy learning only updates from measured reward. A successful HTTP response does not imply that the feedback changed the learner.

Declarative reward components

Each component supports:

Field	Meaning
field	Dot path inside the feedback envelope, such as values.profit
goal	maximize, minimize, or target
min, max	Expected numeric range used to normalize utility to [0,1]
target	Desired value for a target component
tolerance	Distance from target at which utility reaches zero
weight	Relative contribution to weighted or geometric aggregation
hard	Force total reward to zero when this component’s utility is zero
required	Treat a missing or non-numeric value as a measured hard failure

Aggregation modes:

Mode	Use it when
weighted_mean	Objectives can compensate for one another
geometric_mean	A very weak component should strongly reduce the total
minimum	The weakest resolved objective should define the reward

Example with performance and safety:

{
  "aggregation": "geometric_mean",
  "components": [
    {
      "field": "values.throughput",
      "goal": "maximize",
      "min": 0,
      "max": 100,
      "weight": 2
    },
    {
      "field": "values.error_rate",
      "goal": "minimize",
      "min": 0,
      "max": 0.1,
      "weight": 1,
      "hard": true,
      "required": true
    }
  ]
}

Without components, Adapt-1 recognizes numeric values.reward, values.score, or values.utility; Boolean correct, success, or accepted; and recognized binary outcome labels. Fields such as error_distance have no implicit reward meaning. Either declare them as components or compute a normalized reward externally.

Reward scale and neutral reward

Keep rewards in [0,1]. The default neutral point is 0.5:

• below 0.5 is disadvantageous;
• 0.5 is neutral or unresolved;
• above 0.5 is advantageous.

For a naturally signed reward, define normalization explicitly. For example, map profit from [-100,100] with min: -100, max: 100, and goal: maximize.

Do not invent a success or failure label for an unknown outcome. Use an outcome value accepted by the current feedback schema and provide the numeric field required by the reward declaration. Do not rely on undocumented outcome labels; the live validator is authoritative for accepted enum values.

7. Configure sequential return learning

Sequential parameters

Parameter	Default	Effect	Tuning guidance
enabled	false	Enables episode-aware return learning	Must be true for sequential Q learning
episode_path	metadata.episode_id	Groups transitions into episodes	Keep stable across one episode and distinct across episodes
step_path	metadata.step	Orders transitions within an episode	Use monotonic, unique step values
next_context_path	values.next_state	Supplies the observed successor state	Must resolve to an object on every eligible feedback row
reward_path	values.step_reward	Supplies the sequential reward	Must resolve to a finite number
terminal_path	values.terminal	Marks episode completion	Send a real Boolean on every row
discount	0.95	Controls how much future value affects earlier actions	Lower for short horizons; higher for long delayed effects
n_step	3	Number of observed rewards used before bootstrapping	Increase for delayed rewards; reduce for dense/noisy rewards
minimum_episodes	8	Minimum distinct episodes before sequential fitting	The runtime accepts lower positive values, but fewer episodes weaken episode-held-out validation; increase when episodes vary substantially
minimum_validation_skill	0.25	Required held-out improvement over a constant-return baseline	Lower cautiously for noisy data; never use it to force a failing model into selection
target_sync_interval	10	Epoch interval for copying online weights to the target network	Increase for a slower, more stable target; reduce for faster tracking
conservative_weight	0.05	Penalizes unsupported action values	Increase when action coverage is sparse or overestimation is costly

The learner uses episode-held-out validation. The last portion of ordered episode IDs is reserved for validation, with at least two validation episodes. A model is installed only when its validation skill meets the declared threshold. This is why one long episode is not a substitute for multiple episodes.

Picking discount

Use the effective horizon as a guide:

effective horizon is approximately 1 / (1 - discount)

Approximate examples:

discount	Effective horizon
0.80	5 steps
0.90	10 steps
0.95	20 steps
0.98	50 steps
0.99	100 steps

Very high discount values can propagate noise and make policies hard to distinguish. Start from the shortest horizon that can still reach the delayed consequence of interest.

Picking n_step

• Dense reward every step: start at 1 to 3.
• Reward after a short action chain: start at 4 to 8.
• Sparse terminal reward: start at 8 to 20, with eligibility traces enabled.
• Highly stochastic outcomes: use a smaller value and more episodes.

n_step does not need to equal the episode length. It controls the direct return window before the model bootstraps from the successor state.

8. Configure delayed credit

credit_assignment.mode controls synchronous reward attribution in the online policy state. Sequential Q learning separately consumes the episode transitions and returns.

Parameter	Default	Effect
mode	none	Use eligibility_trace to propagate delayed non-neutral outcomes
discount	0.9	Exponential decay applied per delay step
neutral_reward	0.5	Baseline around which positive and negative credit moves
minimum_weight	0.0	Stops backward propagation after the discounted weight falls below this value
delay_path	values.delay_steps	Optional explicit delay used to adjust the current feedback reward

With eligibility traces enabled, a later non-neutral step_reward can update earlier neutral transitions in the same episode. Propagation stops at a prior non-neutral signal, a terminal boundary, an unchanged transition, or the minimum weight cutoff.

Use mode: none when rewards are immediate and accurately assigned. Use eligibility_trace when the effect arrives later than the responsible action.

Do not use a large minimum_weight for long sparse-reward episodes. With discount: 0.95, a cutoff of 0.05 reaches roughly 58 steps; a cutoff of 0.01 reaches roughly 90 steps.

9. Configure exploration and selection safety

Exploration modes

Mode	Behavior	Best use
exploit	Select the highest current expected return	Frozen evaluation and stable deployment
ucb	Adds uncertainty-scaled optimism	Broad, controlled exploration
thompson	Samples from each policy posterior	Stochastic exploration proportional to uncertainty
auto	Chooses among exploit, UCB, and Thompson from confidence and bound-transition status	General online operation

The Domain configuration sets the default, but exploration only occurs when the query sends allow_exploration: true. A query can temporarily override the mode with selection_mode.

exploration_strength scales UCB uncertainty. 0 removes the UCB bonus. Values near 0.5 are cautious, 1.0 is a reasonable start, and values above 1.0 deliberately favor underexplored actions.

Selection gates

Parameter	Default	Effect
minimum_expected_reward	0.0	Abstain if the best candidate is below this expected reward
minimum_confidence	0.0	Abstain if contextual confidence is too low
require_credible_dominance	false	Require the winner’s credible interval to dominate competitors
dominance_margin	0.0	Required separation when credible dominance is enabled
abstain_on_ood	false	Abstain when the context is outside learned support

These gates are deployment controls, not training accelerators. Keep them loose while collecting initial coverage, then tighten them using held-out operational data. The client must always handle status: "abstained"; abstention is a valid safety output.

10. Configure transfer and nonstationarity

Policy transfer

Parameter	Default	Meaning
transfer_strength	0.15	Weight of same-policy evidence learned under another relation
action_transfer_strength	0.05	Weight of evidence from different policies with similar policy_features
min_context_observations	4	Local support target used by confidence and calibration machinery
model_max_weight	0.45	Maximum contribution of the asynchronously trained model
model_ood_threshold	0.75	Novelty distance at which model contribution falls to zero
ood_confidence_penalty	0.5	Confidence reduction under out-of-distribution context

Set both transfer strengths to 0 when actions or relations have unrelated semantics. Increase action_transfer_strength only when policy features have stable physical or operational meaning. Transfer never crosses owner, Domain, or session scope.

Decay

decay_half_life_seconds exponentially reduces the influence of older feedback. 0 disables time decay.

Use decay for regime changes, market drift, changing users, or equipment aging. Pick a half-life in application time, not request count. It should be longer than ordinary noise but shorter than the expected persistence of obsolete behavior.

Examples:

• stable physical process: 0 or several months;
• weekly operational drift: several days to weeks;
• intraday regime changes: tens of minutes to hours;
• controlled reversal test: long enough to retain evidence, short enough for new evidence to dominate.

11. Configure latent belief

Latent belief separates recurring or changing context regimes without requiring the client to provide a regime label.

Parameter	Default	Effect
enabled	true	Enables latent regime inference
max_regimes	8	Maximum regimes retained per relation
novelty_threshold	0.55	Context distance needed to create a new regime
temperature	0.25	Softness of regime probability assignment
change_point_threshold	0.55	Residual threshold for suspected behavior change
change_point_patience	3	Consecutive high residuals required for a change point
historical_regime_weight	0.05	Residual influence of evidence from non-active regimes

Tuning effects:

• Lower novelty_threshold: creates regimes more readily; useful for distinct modes, risky with noisy features.
• Higher novelty_threshold: merges more contexts into one regime.
• Lower temperature: sharper regime assignments.
• Higher temperature: blends evidence across nearby regimes.
• Lower change_point_patience: reacts faster and risks false changes.
• Higher historical_regime_weight: improves recurrence reuse but increases interference from old regimes.

Start with the defaults. Tune latent belief only after verifying that feature scaling, reward, and episode linkage are correct.

12. Configure offline model training

Feedback updates the online posterior synchronously. Neural model fitting is asynchronous and feedback-triggered. The API remains available while training runs.

Parameter	Default	Effect	Guidance
enabled	true	Enables asynchronous model fitting	Disable only when posterior-only behavior is required
min_samples	24	First sample count at which training may start	Use 64 to 256 for noisier sequential tasks
retrain_interval	8	New policy samples required after the last completed fit	Increase to reduce training churn; decrease for fast adaptation
dimensions	96	Capacity control for the contextual model	Sequential Q input capacity is derived from the declared feature schema
epochs	100	Optimization epochs per fit	Increase only if validation is underfit, not merely because training loss is high
learning_rate	0.04	Optimizer step size	Reduce if validation is unstable; increase cautiously if learning is consistently slow

retrain_interval: 20 is not inherently too early for a 2,000-sample task. It means at most one new fit for every 20 accepted feedback samples after the minimum. Choose the interval from the acceptable compute load and adaptation delay:

• rapid drift, cheap fitting: 8 to 20;
• stable task, moderate load: 32 to 100;
• expensive fitting or very large streams: 100 or more.

Training readiness also depends on minimum_episodes. A fit triggered before enough episodes exist may train the ordinary contextual candidate; once enough valid episodes accumulate, the sequential candidate can be trained and validated.

13. Execute the online loop

Step 1: query before acting

curl -sS -X POST \
  "$ADAPT1_BASE_URL/domains/sequential-controller-v1/query" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "ignored",
    "question": "Select the controller for the current state.",
    "relation": "controls",
    "context": {
      "event_type": "state",
      "values": {
        "queue_depth": 71,
        "error_rate": 0.04,
        "capacity": 0.62,
        "operating_mode": "degraded"
      }
    },
    "allow_exploration": true,
    "selection_mode": "auto"
  }'

Persist these response fields:

• decision_id;
• selection.status;
• selection.selected_policy;
• the exact action actually executed;
• the pre-action context.

If the application overrides Adapt-1’s selection, send feedback for the policy that was actually executed. Do not credit an action that was only proposed.

Step 2: observe the transition

After executing the selected policy, collect:

• the successor state;
• the reward components;
• whether the episode ended;
• the episode ID and step;
• any measured delay.

Step 3: write decision-linked feedback

curl -sS -X POST \
  "$ADAPT1_BASE_URL/domains/sequential-controller-v1/feedback" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "ignored",
    "outcome": "success",
    "feedback_kind": "execution",
    "decision_id": "DECISION_ID_FROM_QUERY",
    "relation": "controls",
    "policy": "SELECTED_POLICY_FROM_QUERY",
    "values": {
      "step_reward": 0.65,
      "next_state": {
        "queue_depth": 54,
        "error_rate": 0.025,
        "capacity": 0.61,
        "operating_mode": "recovering"
      },
      "terminal": false,
      "delay_steps": 1
    },
    "metadata": {
      "episode_id": "episode-0001",
      "step": 0
    }
  }'

Repeat query, action, and feedback until the terminal transition. The terminal row still needs a valid next_state object and must send "terminal": true.

decision_id is the preferred binding for a query → action → outcome loop because it uses the sealed pre-action context. It is not the only context route. Current runtime policy admission can also recover context from target_memory_id or from explicit structured context. In every case, send the executed relation and policy explicitly and verify that credit_assignment.contextual_learning_applied is true.

Current admission checks require the relation field to be present and do not validate its value against the bound decision. Send the real relation from the Domain. A successful feedback response without relation can store the record while leaving feedback_policy.sample_count unchanged.

14. Batch feedback without losing sequence semantics

POST /domains/{domain_id}/batch accepts ordered event and feedback operations. Every feedback operation retains its own decision_id, metadata.episode_id, metadata.step, successor state, reward, and terminal flag.

Batching reduces HTTP overhead; it does not remove the need for per-transition identity. It is safe when decisions have already been made and their results are being uploaded together. It cannot replace the online query loop when the next action depends on the previous query result.

Minimal shape:

{
  "session_id": "ignored",
  "operations": [
    {
      "operation": "feedback",
      "ref": "feedback-episode-1-step-0",
      "outcome": "success",
      "feedback_kind": "execution",
      "decision_id": "decision-step-0",
      "relation": "controls",
      "policy": "balanced",
      "values": {
        "step_reward": 0.5,
        "next_state": {"queue_depth": 62, "error_rate": 0.03, "capacity": 0.62, "operating_mode": "degraded"},
        "terminal": false,
        "delay_steps": 0
      },
      "metadata": {"episode_id": "episode-0001", "step": 0}
    },
    {
      "operation": "feedback",
      "ref": "feedback-episode-1-step-1",
      "outcome": "success",
      "feedback_kind": "execution",
      "decision_id": "decision-step-1",
      "relation": "controls",
      "policy": "balanced",
      "values": {
        "step_reward": 0.9,
        "next_state": {"queue_depth": 30, "error_rate": 0.01, "capacity": 0.65, "operating_mode": "normal"},
        "terminal": true,
        "delay_steps": 0
      },
      "metadata": {"episode_id": "episode-0001", "step": 1}
    }
  ]
}

15. Read the diagnostics

Request a normal Domain query and inspect:

Output	What it tells you
selection.status	selected or abstained
selection.reason	Why no policy was selected or which gate applied
selection.selected_policy	Executable action identity
ranked_hypotheses[].contextual_policy.expected_reward	Fused expected immediate/contextual reward
ranked_hypotheses[].contextual_policy.selection_expected_reward	Value actually preserved for policy ordering, including sequential return
ranked_hypotheses[].contextual_policy.sequential_expected_reward	Sequential model estimate when installed
ranked_hypotheses[].contextual_policy.model_weight	Effective trained-model contribution
ranked_hypotheses[].contextual_policy.out_of_distribution	Whether context novelty exceeded support
ranked_hypotheses[].contextual_policy.effective_observations	Weighted evidence supporting the estimate
latent_belief	Current regime probabilities and novelty
learning_state.subsystems.feedback_policy.sample_count	Accepted policy-learning observations
learning_state.subsystems.feedback_policy.model.status	waiting, running, installed, failed, or unavailable state
learning_state.subsystems.feedback_policy.model.report.model_type	Look for sequential_q_mlp after successful sequential fitting
learning_state.subsystems.feedback_policy.model.report.validation_skill	Held-out improvement over the constant-return baseline

The top-level learning_state.sample_count is an aggregate across active learning subsystems. Use the feedback-policy subsystem count when diagnosing sequential policy learning.

Do not wait for a separate sequential subsystem to appear. Sequential return learning is represented by the model lifecycle inside learning_state.subsystems.feedback_policy. Before feedback_policy.sample_count increases, the episode, sample, training, and validation thresholds for sequential_q_mlp are not yet relevant. Samples admitted by the separate structured_transition subsystem do not satisfy the policy learner’s sample or episode gates.

Evidence that sequential learning is active includes all of the following:

1. feedback-policy sample_count increases after accepted feedback;
2. multiple valid episode IDs are retained;
3. model status advances from waiting to running and then installed;
4. the installed model type is sequential_q_mlp;
5. validation skill meets the configured threshold;
6. sequential expected rewards differ across policies or contexts;
7. frozen exploit decisions improve on held-out episodes without feedback writes.

16. Freeze evaluation correctly

To measure learned behavior rather than continued adaptation:

1. train only on the training partition;
2. wait for asynchronous training to complete;
3. stop all /feedback writes;
4. query held-out episodes with allow_exploration: false;
5. use selection_mode: "exploit";
6. keep the same authenticated owner and domain_id so the trained state remains available;
7. use unseen episode IDs, and preferably unseen scenarios or entities;
8. record abstentions as abstentions rather than silently replacing them in the Adapt-1 score.

Frozen query example:

{
  "session_id": "ignored",
  "question": "Select the controller for this held-out state.",
  "relation": "controls",
  "context": {
    "event_type": "state",
    "values": {
      "queue_depth": 83,
      "error_rate": 0.052,
      "capacity": 0.58,
      "operating_mode": "degraded"
    }
  },
  "allow_exploration": false,
  "selection_mode": "exploit"
}

Report at least return, policy accuracy if ground truth exists, abstention coverage, selective performance, episode variance, and learning curves by episode. Use multiple seeds or independent streams for stochastic environments.

Advanced configuration

17. Tuning profiles

These are starting profiles, not guaranteed optima.

Sparse delayed reward

{
  "context": {"max_samples": 4096},
  "credit_assignment": {
    "mode": "eligibility_trace",
    "discount": 0.97,
    "neutral_reward": 0.5,
    "minimum_weight": 0.005
  },
  "sequential": {
    "discount": 0.98,
    "n_step": 12,
    "minimum_episodes": 12,
    "conservative_weight": 0.08
  },
  "training": {"min_samples": 128, "retrain_interval": 32}
}

Use when most steps are neutral and terminal outcomes carry the useful signal.

Dense short-horizon control

{
  "credit_assignment": {"mode": "none"},
  "sequential": {
    "discount": 0.9,
    "n_step": 3,
    "minimum_episodes": 8,
    "conservative_weight": 0.03
  },
  "training": {"min_samples": 64, "retrain_interval": 16}
}

Nonstationary or reversal-prone task

{
  "policy": {
    "decay_half_life_seconds": 86400,
    "transfer_strength": 0.0,
    "action_transfer_strength": 0.05
  },
  "latent_belief": {
    "enabled": true,
    "max_regimes": 12,
    "novelty_threshold": 0.45,
    "change_point_threshold": 0.4,
    "change_point_patience": 2,
    "historical_regime_weight": 0.1
  },
  "training": {"retrain_interval": 8}
}

Set the decay half-life in real application time. 86400 is one day and is only an example.

High-stakes conservative deployment

{
  "policy": {
    "exploration_mode": "exploit",
    "minimum_expected_reward": 0.6,
    "minimum_confidence": 0.4,
    "require_credible_dominance": true,
    "dominance_margin": 0.05,
    "abstain_on_ood": true
  },
  "sequential": {
    "minimum_validation_skill": 0.25,
    "conservative_weight": 0.12
  }
}

Collect exploration data in a safe environment before enabling these deployment gates.

Many structurally similar actions

Use informative policy_features, set action_transfer_strength around 0.1 to 0.25, and increase conservative_weight when many actions remain unsupported. Verify with an ablation against action_transfer_strength: 0 to confirm transfer helps held-out actions rather than blending distinct ones.

18. Advanced bound-transition projection

learning.sequential.bound_transition is an optional non-neural projection for tasks with explicitly bound state objects and action roles. It can estimate how a declared transition action changes a bounded objective and blend that estimate into policy scoring. Most sequential tasks should leave it disabled.

It is appropriate when:

• one context contains multiple indexed entities or objects;
• a policy selects which entity is the current state and optionally which is the action source;
• action roles can be declared from policy features;
• the objective has known numeric bounds;
• state transformations can transfer between structurally similar bindings.

Parameter	Meaning
enabled	Enables bound-transition projection
state_selector_feature	Policy-feature path whose value selects the current state object
state_features	Map from logical feature name to context path template; {selector} is substituted
state_bounds	Optional [min,max] validation bounds by logical state feature
action_selector_feature	Optional policy-feature path selecting an action/source object
action_features	Map from logical action feature to templated context path
signature_features	Policy-feature paths that must match for direct transition evidence
objective_feature	Logical state feature to maximize after normalization
episode_start_path	Optional context Boolean that resets episode-local binding
objective_min, objective_max	Numeric normalization limits
transition_values	Policy-feature path to allowed values identifying transition actions
terminal_values	Policy-feature path to allowed values identifying terminal actions
minimum_support	Required matching transitions before prediction
model_weight	Blend weight of the bound projection in policy scoring
novelty_bonus	Exploration value assigned to unsupported transitions
transition_cost	Utility subtracted from predicted transition actions

This surface is for explicit, reusable structural bindings. Do not enable it merely to increase a benchmark score. If the application cannot state what the selectors, state features, action roles, and objective mean independently of its evaluation labels, use the ordinary sequential learner.

19. Common failure modes

sample_count stays at zero

First distinguish storage from learner admission. A successful feedback response can store the record while leaving feedback_policy.sample_count unchanged.

Check:

• learning.enabled is true;
• feedback includes a real measured reward;
• relation is present in the feedback request;
• the executed policy is present and is one of the declared hypotheses;
• decision-time context is recoverable from a valid decision_id, a valid target_memory_id, or explicit structured context.

Then inspect credit_assignment.contextual_learning_applied. If it is false, the record did not enter contextual policy learning even if the HTTP request succeeded. Current responses report the context route under credit_assignment.context_source as decision, memory, or request.

An unrecognized outcome label does not update policy learning unless a numeric, Boolean, or declarative reward is also present.

Feedback count rises but no sequential model appears

Check every row for all five configured sequential paths and types. Then check:

• at least minimum_episodes distinct episode IDs exist;
• replay capacity retains enough rows from those episodes;
• training has reached min_samples;
• Torch or the configured training service is available;
• asynchronous fitting has completed;
• validation skill meets minimum_validation_skill.

Every decision abstains at the start

Use allow_exploration: true during data collection. With no policy evidence and exploration disabled, abstention is expected. Keep strict confidence, dominance, and OOD gates disabled until there is enough coverage.

Decisions return selected_policy: null

Only declared hypotheses with non-null policies are executable. Predictive or induced hypotheses without a policy cannot be selected as actions. Verify that all competing policy hypotheses share the requested relation and have unique policies.

Learning becomes worse across episodes

Check for:

• reward direction or normalization errors;
• incorrect decision-to-feedback linkage;
• outcome leakage in context features;
• reused episode IDs or non-monotonic steps;
• too-small replay capacity;
• excessive transfer between unrelated actions;
• too-high discount or n_step for noisy rewards;
• validation episodes drawn from a different regime due to lexicographic episode naming;
• exploration still enabled during evaluation;
• client fallbacks being scored as if Adapt-1 selected them.

The model trains but policy ordering stays flat

Inspect validation_skill, sequential_expected_reward, model weight, and OOD status. Common causes are constant rewards, action-independent features, identical policy features, low action coverage, a feature path mismatch between current and successor state, or OOD downweighting.

20. Building the learner for a new use case

Use this sequence:

1. Define the decision. Write one sentence: “Given state X, choose one policy from Y to maximize measured outcome Z over horizon H.”
2. Declare policies. Give every executable action a unique policy and observable policy features.
3. Declare pre-action state. Select only fields known before acting.
4. Define reward mathematically. Normalize every objective to [0,1]; specify tradeoffs and hard constraints.
5. Define episode boundaries. Establish stable episode IDs, ordered steps, successor state, and terminal semantics.
6. Estimate the horizon. Pick the shortest discount and n_step that reach the relevant consequences.
7. Size replay. Retain enough transitions from enough complete and diverse episodes.
8. Choose exploration. Explore in a safe environment; exploit or abstain in high-risk deployment.
9. Set validation gates. Require positive held-out skill and keep conservative penalties when action coverage is incomplete.
10. Instrument the loop. Persist decisions and verify that every executed action receives correctly linked feedback.
11. Run ablations. Compare sequential enabled/disabled, transfer enabled/disabled, and latent belief enabled/disabled.
12. Freeze and evaluate. Use unseen episodes with no feedback writes and no exploration.

21. Pre-deployment checklist

• The authenticated owner and Domain stay stable across training episodes.
• Every policy is unique, executable, and attached to the queried relation.
• Policy features describe actions, not outcomes.
• Context features contain no post-action or answer information.
• Reward is measurable, normalized, and directionally correct.
• Every feedback row has episode, step, next state, step reward, and Boolean terminal.
• Every executed action has one valid decision-time context binding: sealed decision_id, valid target_memory_id, or explicit structured context; use decision_id for sealed query attribution when available.
• Replay capacity covers several representative episodes.
• Exploration is enabled only where exploration is acceptable.
• The client handles abstention explicitly.
• A sequential_q_mlp is installed only after positive held-out validation skill.
• OOD and confidence behavior is tested before high-stakes deployment.
• Evaluation is frozen, held out, and free of feedback writes.
• Results include variance, coverage, abstention, and learning curves rather than only the best episode.

The strongest sequential learner is produced by a clean data contract and a defensible evaluation protocol. Parameter tuning cannot compensate for missing successor states, ambiguous reward, cross-session fragmentation, or incorrect action-feedback linkage.

Sequential Discovery

Start with the Discovery behavior, public boundary, and minimal contract.

Discovery overview

Review Transition, Structure, and Sequential Discovery together.