Structured Decision and Safety Models

Bound the answer, not the truth

A decision model receives state, a question and a bounded answer space. It returns an option or a distribution that ordinary code can consume. This avoids asking a general chatbot to invent both an answer and its serialization.

The constraint removes some failure modes, not all of them. Selecting an existing product ID prevents an invented ID. The selected product can still be irrelevant. Selecting an allowed action does not authorize the application to execute that action.

Keep arithmetic, permissions and workflow invariants in code. Use a model where semantic judgment is genuinely difficult to express as an exact rule. The same boundary applies to routing, data curation and moderation.

Choose the decision primitive

Jev is TypeSafe's decision product, not an acronym for a published architecture. Its System One label emphasizes quick bounded judgments. Together's Tev is a separate open Jev-inspired baseline, not a reproduction of the private model.

The question conditions the meaning of each answer slot. Options can change across requests. A question ID is application bookkeeping, not model-visible semantic instruction. Questions in one request are documented as independently evaluated against the same state.

Use descriptive levels for Score, not arbitrary numbers without a rubric. A score between levels is a position on that rubric, not an exact physical measurement. TypeSafe primitives ↗ defines the interface.

Read the distribution before its summary

All mass on level 1 and equal mass on levels 0 and 2 both have mean 1. The first is concentrated; the second is split across extremes. A downstream system that uses only the mean loses that distinction.

Ordered levels impose a geometry that unordered options do not have. A near-neighbor disagreement can be operationally different from disagreement across opposite ends. Still, the level spacing comes from the rubric; it is not automatically an interval scale of real-world harm.

Confidence summarizes a distribution, not independent evidence

As checked on 2026-10-06, TypeSafe now publishes the formulas. The older September note's statement that the exact formula was unspecified is no longer current. These are deterministic summaries, not a second verifier.

Choice uses the top mass relative to a uniform split. Score also uses ordered distance from the mode. The interactive examples apply those definitions; their distributions are assigned, not measured Jev outputs.

Noul returns its yes probability without a separate confidence field. A derived binary summary |2p−1| is possible, but is not the yes probability itself. A sharp wrong distribution can be confident. Validate thresholds on outcomes before treating any summary as an automation guarantee. Current confidence definitions ↗

Train probabilities and decisions for the right target

A proposed calibrated classifier first needs semantic discrimination. Train state/question/answer-space tuples with hard-label cross-entropy or reliable soft targets. Human disagreement, teacher probabilities and objective event frequencies are different kinds of targets.

The proper scoring objective rewards reporting the distribution, not merely selecting the more likely action. Its population optimum does not guarantee calibration with finite data, misspecified models or distribution shift. This example does not reveal Jev's undisclosed RLCD objective.

Randomize option order and remap targets. Vary phrasing without changing the label. Include realistic ambiguity, missing evidence, hard negatives and abstention where useful. Split by source document, customer, repository or template family; otherwise near-identical views can leak across the holdout.

Keep exact arithmetic outside the target model. Add adversarial state examples while preserving trusted instructions as a separate boundary. Only add RL when downstream utility, information acquisition or sequential control gives it a defined role. A static classification task does not automatically need GRPO.

Measure uncertainty on a separate holdout

Reserve a calibration set and an untouched final test set. Measure log loss, Brier score, reliability diagrams, task accuracy and selective risk. Selective evaluation asks how much traffic can be automated at an acceptable error rate.

Temperature scaling fits one positive temperature on held-out logits. Dividing logits by that temperature changes concentration without changing the argmax. It cannot fix missing classes, poor features or arbitrary deployment shifts.

Measure calibration by domain, language, policy and option count. A pooled reliability curve can conceal a severe failure in one subgroup. When prevalence changes, the relationship between score and outcome can also change.

Choose an action threshold using observed costs and coverage. A classification probability is not permission, and a probability-weighted mean is not a decomposition of epistemic versus inherent uncertainty. Losses and Objectives explains proper scoring; Guo et al. (2017) ↗ studies temperature calibration.

What Jev discloses and what remains unknown

TypeSafe's launch discloses a parallel sampler and Reinforcement Learning for Calibrated Decisions (RLCD). It describes outputs produced in parallel rather than sequential prose generation. Public material reviewed here does not disclose a layer diagram, parameter count, checkpoint lineage, training-data volume or implementable RLCD loss.

Do not infer a particular backbone, attention mask or policy-gradient method from the product interface. The three designs below are engineering alternatives, not reverse-engineered Jev internals. A pretrained semantic representation is a useful starting point, but that does not identify the private architecture.

The vendor describes roughly 70–500 ms response times and large workflow advantages under selected conditions. Short inputs and reasoning-model comparisons favor the demonstration. These are vendor measurements, not universal speedups over conventional classifiers. TypeSafe launch ↗ supplies the claim and its caveats.

Baseline A: reuse the language-model head

The simplest baseline reuses a language model's output head. Supply the state, question and option descriptions. At the answer position, take logits for allowed labels and normalize within that set.

Check that each label is one token under the exact template, whitespace and tokenizer. Multi-token labels require a different scoring contract. Restricting the denominator gives a distribution conditional on the supplied set; it does not ensure that the set contains a correct answer.

A one-token decision already avoids long autoregressive output. Include this baseline in performance comparisons. Constrained decoding alone does not provide calibrated probabilities or semantic correctness.

Design B: score dynamic options explicitly

A second proposed design adds a small decision head to a pretrained backbone. The prompt supplies the meaning of dynamic option slots. Mask unused slots before softmax, with at least one valid slot required.

h = backbone(state, question, option_descriptions)
logits = decision_head(h)  # [B, maximum_slots]
logits = mask_invalid_slots(logits, valid_slots)
probabilities = softmax(logits.float(), dim=-1)

Another design scores each option with the same function score(state, question, option). Shared scoring accommodates dynamic descriptions and can support permutation-equivariance, but may repeat input computation without caching.

A binary head can implement yes/no; an ordered head can implement rubric levels. Removing a large vocabulary projection may reduce head cost. It does not remove input processing or automatically calibrate the result. Test option permutations, masks and an explicit unknown option.

Design C: share state without mixing question branches

A third proposed design computes expensive state features once, then isolates question branches. Cross-attention over shared encoder features and a causal shared-prefix KV cache are different mechanisms. Neither is confirmed as Jev's implementation.

For state cost 100, question cost 5 and overhead 10, eight independent calls cost 840 units; shared-state execution costs 150. These are assigned operation units, not measured latency or billed tokens.

Branches must not read one another's answers if independent evaluation is part of the API contract. Parallel question execution does not make the judged real-world events statistically independent. Questions needing new tool evidence still require another stage.

Speculative fan-out asks several potentially needed questions together to reduce network round trips. That spends extra work on unused branches. It is unrelated to speculative token decoding's acceptance proof.

Tev: a concrete reproducible baseline

Tev1-4B-experimental uses ordinary LoRA SFT on Qwen3.5-4B with its existing language-model head. The repository does not train on Jev-generated answers. It supports two to twenty-four options.

The final export has 4,568 development examples, 16,929,529 training tokens and maximum exported sequence length 1,526. It removes 6,000 exact replay duplicates and checks split overlap, conflicting duplicates, template alignment and answer/EOS formatting.

Synthetic research records are templated exercises, not thousands of independently observed papers. The tutorial's approximate 38,000/38,340 statements differ; use the precise saved-dataset count.

The current dataset guide identifies the pinned Qwen3.5-2B tokenizer as the historical export-format pin, not the Qwen3.5-4B training backbone. Preserve template bytes separately from checkpoint identity. Dataset guide ↗

Keep defaults separate from historical job settings

The target is an answer letter followed by EOS. The application maps that letter to a semantic key. The student is not trained to generate the final JSON result object. Match training and inference templates, including non-thinking mode.

Retain uploaded file identity, tokenizer pin and actual job settings. A documented default is not proof that the saved checkpoint used every default. Training guide ↗ · Training example ↗

A letter target is not generated JSON

state:   The user asks for last month's invoice.
question: Which team should handle this request?
A: billing — invoices and payments
B: technical — product failures
C: unknown — insufficient information
completion: A + EOS

This is an illustrative record, not a copied dataset row. The inference example disables thinking, uses temperature 0, allows up to 8 output tokens and constrains labels with a regex. It requests log probabilities with only five top alternatives.

With more than five choices, top-five output is not necessarily a full distribution over allowed labels. Missing probability mass matters. Token preferences do not become calibrated correctness probabilities because the selected letter is valid.

Validate the output and map only known letters to supplied keys. Treat invalid or incomplete output as an explicit failure. Inference example ↗ is the implementation reference; SFT explains target masks and template compatibility.

A small training bill is not product equivalence

The tutorial reports about $17 of fine-tuning cost and 25 minutes for one run. This excludes pretraining, product development, dataset design and continuing endpoint cost. Its H100 80GB endpoint is a serving example, not evidence of the historical training hardware.

Saved results show 880/1,000 main decisions and 300/300 policy-transfer decisions. The repository labels them reused development benchmarks, not untouched final tests. They do not establish equality with Jev's accuracy, calibration or architecture.

An inexpensive baseline is valuable because it creates a concrete alternative to test. After using the evaluation to tune data or settings, collect a fresh holdout. The training bill alone says little about the cost of achieving a reliable application.

Together tutorial ↗ · Run evidence ↗ · Current repository ↗

Compose small judgments with exact code

For a support ticket, ask separately whether evidence is sufficient, whether a refund is requested and which category applies. Code checks deadlines and permissions. Combining independent semantic questions with deterministic logic makes errors easier to inspect.

Cheap judgments can inspect every retrieved item or agent step. Larger evaluation volume can also amplify systematic mistakes. Measure the resulting workflow, not only isolated classifier accuracy.

Extract candidates, select, then copy

A provenance-preserving extraction pipeline first enumerates candidate values through a parser, regex or a generative extractor. A semantic classifier selects among them. Code copies and normalizes the selected original value.

Keep source spans and a not-stated option. Dates are a useful example: the model can help identify mentioned components; code validates and compares real dates. Do not use a rubric mean to reconstruct an exact numerical quantity.

Jev1.13's documentation reports counting, numerical precision, dates, indirection, irrelevant context and adversarial-state weaknesses. Separately asked propositions can also disagree with logical invariants; option order can affect the answer. These are version-specific limitations, not guarantees about every future release. Jev1.13 limitations ↗ recommends keeping exact operations in code.

Measure the confident errors that never escalate

A fast/slow cascade uses a cheap classifier for well-covered cases and escalates uncertain or unfamiliar inputs. A stronger reasoner or a reviewer handles the remainder. Thresholds must be chosen from outcomes, not vendor confidence terminology.

A confident error that passes the gate never reaches the review queue. Evaluating only escalated requests misses that failure. The example exposes this directly: a high-score incorrect decision remains automatic under many thresholds.

A stricter gate reduces coverage; it need not monotonically reduce empirical selective risk on a finite dataset. Review outcomes are not simulated here. An empty automatic set has undefined selective risk, not zero demonstrated error. Permissions remain enforced outside the classifier.

Benchmark the complete bounded workflow

Compare exact rules where appropriate, conventional encoder classifiers or rerankers, prompted one-token small LLMs, a Tev-style fine-tune, Jev and a stronger reasoning model. Use the same state, answer schema, batching opportunity and quality target.

Report p50/p95 latency, input size, number of questions/options, throughput under load, calibration, abstention coverage and total cost. Disable unnecessary reasoning for a non-reasoning baseline. A long generated distribution is not the only alternative to a decision API.

TypeSafe's workflow evaluation uses a fixed code graph and probabilities from strong external models as references. Agreement with those references is not independent ground-truth correctness, particularly on ambiguous judgments. Workflow methodology ↗ states the evaluation target.

A typed interface can improve integration independently of model quality. Measure each benefit separately instead of treating a clean schema as proof of superior intelligence.

Make the policy part of the input

Safety taxonomies differ in scope and granularity. Deployment context also changes what the classifier must check. A fixed output taxonomy cannot express every operator policy without additional logic, adaptation or a separate model.

Shieldstral makes moderation a binary question-answering task. The instruction specifies context and strictness. The query asks one criterion. The document can be a prompt, response, formatted dialogue, image or image/text pair.

The important contribution is this conditional formulation and its data construction, not a new 3B backbone. A no answer to one category does not mean the document is globally safe. The policy must still be coherent, authorized and operationally testable.

Separate vocabulary training from binary inference

Training uses ordinary cross-entropy over the full vocabulary at the single output position. Inference projects toward yes/no token IDs and normalizes those two values. Full-vocabulary training and restricted binary normalization are different contracts.

One decision forward pass avoids a long generated explanation. Input processing can still dominate cost, especially with long text or images. The bounded score is not automatically calibrated across language, modality, policy or prevalence.

Check token IDs, prompt tags, position and exact threshold handling. Inverse wording changes what yes means; translate that answer back into policy semantics before enforcement.

Constructed views are not unique documents

The sources include safety, toxicity, hate, jailbreaks, moderation, refusal detection and response quality. They supply heterogeneous supervision, not one universal binary taxonomy.

Several questions per document, repeated positives, paraphrases, ancestor labels and visual query mutations expand the count. Those views can improve training, but 54.1M views are not 54.1M independent semantic documents.

Track unique content, view construction and sampling weight separately. Repeated positive views are an optimization choice, not newly observed evidence. Training Data explains how token mass and source mixture interact.

Unify tasks without erasing label semantics

A manually designed processor interprets each source's native labels and defines its positive/negative conditions. It maps categories, document format, task instruction and query pools. An LLM generates paraphrases of that explicitly defined mapping.

Strict is an input feature, not a certified false-positive rate. Validate each wording on deployment data. Category, overall-safety and refusal queries have distinct target meanings.

Document variants include plain role labels, brackets, XML-like tags, Markdown, conversational wording and minimal delimiters. They reduce dependence on one serialization. For multimodal examples, ask separately about image, text and combined content; define combination logic explicitly.

Hold content fixed; change the question

A generally harmful document can be a positive example for one policy and a hard negative for another. Hold the document fixed and ask whether the particular criterion applies. This prevents the shortcut of answering yes whenever any generic harmful signal appears.

Public-data positives include general unsafety, the annotated category and relevant target groups. Negatives use absent categories, unrelated demographics and genuinely safe content. Nearby sibling categories create harder contrasts than unrelated categories.

Many absent categories can produce more negatives than positives. Shieldstral duplicates positives with independently sampled instruction/query paraphrases to rebalance. A verifier LLM removes examples when its binary or per-category classification disagrees with the source annotation.

Agreement filtering may improve consistency while importing verifier biases. Ambiguous or culturally variable examples can be disproportionately removed. Keep filtered-out cases for analysis instead of assuming every disagreement was source noise.

Generate fine boundaries and ancestor positives

The synthetic training taxonomy has 11 superclasses and 73 leaf categories derived from 11 source taxonomies. Given a safe source and target/sibling pair, the generator writes content matching the target while avoiding the sibling.

The same generated document receives a positive target query and a negative sibling query. Ancestor queries add positives at broader levels. One generation can therefore supervise several granularities without representing several independent documents.

The reported generation temperature is 0.7. Rewrites preserve the source language while generated queries are in English. That language split is a data contract, not evidence that every multilingual policy wording generalizes equally.

Sibling negatives train fine boundaries: the answer must depend on the requested harm type. Random negative categories may teach only coarse discrimination. This pattern can be adapted to routing, quality assurance and rubric evaluation, but those are proposed transfers, not tested Shieldstral results.

Ground each query in its actual modality

Unsafe image data is scarce. The 4.5M multimodal views combine violation datasets, clean negatives and general classification/detection sources. An LLM mutates queries rather than generating every image.

About 2,000 query phrasings cover a fixed 14-subcategory visual taxonomy spanning NSFW, violence, hate and illegal content. Approximately 30% use inverse wording. That discourages equating a harmful keyword in the question with a yes target.

A vision-language reranker checks image/query labels, query mutations, hallucinated assignments and hard negatives. Filtering is more permissive for scarce violations and stricter for abundant negatives. This preserves rare positives but can retain more positive-label noise.

Separate image-only, text-only and combined policies. A model can succeed by reading accompanying text without understanding the image; evaluate that shortcut. The broader free-form adaptability evidence is stronger for text than for this fixed visual taxonomy.

Change the evaluation ontology, not just label names

The evaluation ontology has 12 superclasses, 26 subcategories, 52 leaves and 90 fixed manually written queries. It changes names, grouping and granularity instead of simply reusing the training labels.

Ten of twelve evaluation superclasses loosely correspond to training domains, while no leaf maps one-to-one. This tests policy-boundary transfer within related safety domains, not arbitrary novel policy understanding.

Evaluation holds the query fixed and changes content: one document matches the target; its contrast matches a sibling but not the target. This iso-query test complements iso-content training. Different generation/verifier models and seed sources reduce direct leakage.

A separate validation set uses the training-generation models and supports ablations. Keep validation and final test results separate. Independent generators can still share synthetic style or taxonomy assumptions; add real ambiguous deployment cases before claiming operational generality.

What the classifier training recipe actually specifies

Shieldstral starts from Ministral-3B-Base-2512 with native multimodal support through a Pixtral vision encoder. It applies LoRA to language-model parameters and cross-entropy to the one-token target.

Each method wins one validation comparison. This supports the selected efficient adaptation in this setup, not the claim that LoRA universally matches full tuning. A reported lack of significant overall advantage is not a formal equivalence guarantee.

The paper does not disclose a complete reproduction recipe: LoRA rank/alpha/dropout/targets, optimizer, learning rate, batch, sequence-length distribution, epochs/consumed tokens, exact source weights, hardware/compute, vision-update details and calibration construction remain unspecified.

The data mechanism is more fully described than optimization. Do not fill these gaps with Tev defaults; the models and targets differ. SFT and Adaptation explains the general LoRA tradeoff.

Preserve complementary checkpoints without assuming linear mixing

P trains on public safety data. PG adds generated taxonomy data. P fits established benchmark behavior better; PG learns finer category distinctions while shifting the distribution. The final recipe preserves both and adds an instruct checkpoint I.

These coefficients describe the report's pairwise spherical linear interpolation (SLERP) configuration. The plus notation must not be read as a plain weighted tensor average. Pairwise order, parameter handling and normalization need an explicit implementation contract.

The observed complementarity motivates retaining specialized solutions. It does not prove that these weights transfer to other backbones or mixtures. Joint multi-objective training is another baseline to test. The instruct component is intended to restore general instruction-following behavior.

Read aggregate performance and ablations together

The paper compares ten guard-model baselines across text, multilingual, refusal, adaptability and multimodal tasks. Its 3B backbone is smaller than all compared text guards. Guard baselines differ in output style and reasoning settings. A single-token 3B model's output can be cheaper than a long guard trace, but the report does not provide production latency, throughput, batching or cost measurements.

The unadapted base defaults to safe. Public data yields conservative high precision with low recall. Generated taxonomy data raises recall sharply while lowering precision. That movement matters as much as the F1 gain.

The 84.4/88.7 validation results and 91.3 test adaptability result are different evaluations; do not place them on a single training curve. Aggregate F1 also does not specify the severity or deployment cost of individual errors. Shieldstral report ↗ is the primary reference.

Keep scope, calibration and reproducibility limits visible

Contradictory criteria cannot be resolved reliably by a type constraint. A policy is itself a versioned artifact to validate. More strict wording can change behavior without a known error-rate change.

A classifier that reports high confidence can still reflect a wrong training label, a missing option or an adversarial state. Keep uncertainty, outcome evidence and enforcement responsibility separate.

A guard is one component of policy enforcement

A moderation service first selects and validates an authorized policy. The classifier then evaluates content against that policy. A separate controller applies calibrated thresholds, permissions, escalation and deterministic fallbacks.

Version policy prompts and templates. Prevent untrusted content from replacing trusted instructions. Record scores and decisions for audit and appeal. Test policy injection, obfuscation, multilingual edge cases and image/text attacks.

Measure false positives and false negatives separately by severity and domain. Decide fail-open or fail-closed behavior by risk class before an outage. A guard timeout must not silently become a safe decision.

Evaluate input and output guards together: filtering a request can change the generator's response distribution, and the output guard may face new content. Monitor score/content drift and periodically refresh calibration. Agents explains executable evidence and verifier attack surfaces.

Two small contracts to implement first

The first kernel is stable binary normalization from raw logits. It does not calibrate the score; it only computes the declared probability within two labels.

margin = yes_logit.float() - no_logit.float()
score = sigmoid(margin)
label_yes = score >= threshold
policy_label = interpret_query_polarity(label_yes, query)
action = controller(policy_label, permissions, budget)

Do not flip a score merely because a query contains the word free. Polarity must come from the authorized query specification. Distinguish a no category match from overall permission.

The second kernel maps a constrained letter to an existing candidate. A selected candidate carries its original source span and deterministic value.

letter = bounded_classifier(state, question, options)
if letter not in options: return explicit_failure()
candidate = options[letter]
if candidate.key == 'unknown': return abstain()
value = normalize(candidate.original_value)
return value, candidate.source_span

Test allowed labels, tokenizer/template alignment, empty option sets, duplicates, unknown results and invalid states. For probability objectives, test mass normalization, masks and loss reduction. For cascades, test automatic error counts, review rates and zero-coverage behavior before adding expensive serving machinery.

Related ideas, different implementation claims

Classification and routing existed before these products. The useful progression is a cleaner question-conditioned interface, explicit evidence boundaries and improved data construction. Choose the simplest measured implementation that satisfies the actual workflow.

Sources and connected topics

Shieldstral ↗ · TypeSafe introduction ↗ · Primitives ↗ · Confidence definitions ↗ · Jev1.13 limitations ↗

Jev launch ↗ · Workflow evaluation ↗ · Together tutorial ↗ · Tev repository ↗ · Calibration of modern neural networks ↗

Additional implementation references. Score rubric semantics ↗ · Training primer ↗ · Speculative fan-out ↗ · Candidate extraction ↗ · Dataset guide ↗ · Training guide ↗ · Training script ↗ · Inference script ↗ · Saved run evidence ↗

SFT · Training Data · Objectives · Statistics and Evaluation · Agents · Distillation · Reasoning

All four interactive explanations use assigned data or exact probability identities. No private model runs, actual moderation actions or production performance are simulated as measured evidence.