Apertus 1.5 opens the recipe, including the parts that did not work

Apertus 1.5 is interesting because its authors show how a text model becomes a multimodal model, and where that process fails. The release includes 8B and 70B checkpoints, image and audio input, a longer context and optional thinking. Its technical report makes the relationships between those features unusually visible: larger is not always better, alignment can worsen spatial precision, and a tool-capable template does not make a reasoning agent.

The most useful way to read this release is as a set of inspectable engineering decisions. Model weights tell a developer what can run. A training recipe helps explain why it behaves that way. Intermediate checkpoints, data inventories and evaluation details make it possible to ask whether a failure appeared during data preparation, continued training, supervised instruction tuning or later alignment. That is a materially richer starting point than a single final score.

This article explains and adapts technical details from the Apertus 1.5 report, published under Creative Commons Attribution 4.0, with attribution to its authors. The interpretations and worked examples below are our analysis. We have not independently trained or benchmarked the models. The October report and the currently available release artifacts should also be distinguished from older text remaining in model cards about a forthcoming report.

Multimodality enters through the vocabulary

Many multimodal systems connect a language model to a separately trained vision encoder and a projection module. Apertus takes a different route. It converts images and audio into discrete codes, adds those codes to the tokenizer’s vocabulary and lets the language model process the resulting sequence. The image and audio tokenizers operate outside the language model’s forward pass. Their codes enter through the same embedding machinery used for the expanded token stream.

That choice preserves the underlying language model architecture while changing what its input sequence can represent. The report describes an image tokenizer derived from Emu3.5 and an audio tokenizer based on WavTokenizer. The language model can encounter mixed text, image and audio sequences through a common next-token training objective. It can also learn from unpaired media before paired examples teach relationships between a signal and its textual description.

There is an important asymmetry. Input vocabulary expansion does not mean the model is released as an image or speech generator. Its generated output is restricted to text. A developer can ask about an image or a recording, but should not describe the release as an image creation system or a conversational voice model with native spoken replies. Input understanding and output generation demand different training, decoding and evaluation.

This design also changes what a token budget means. A token is no longer synonymous with a short piece of written language. It can stand for a small visual region or a slice of compressed audio. An application that bills or truncates according to a text-only mental model can therefore misunderstand how much context a document or recording actually consumes. The useful unit becomes the full preprocessed sequence, including modality markers and task instructions.

The 8B model card documents interleaved inputs and text output, while the 70B card supplies a separate serving configuration. Both matter because an architecture description does not specify a runnable deployment. The cards identify release-specific runtime branches, parsers and memory settings; operators should preserve those version details rather than assume an arbitrary installed package already implements the interface.

An image and a minute of audio have different costs

The report’s image path maps a 16-by-16 pixel patch to one discrete image code. For a 512-by-512 image, that gives a 32-by-32 grid, or 1,024 image codes before structural markers. This is a useful explanatory calculation, not a claim about every possible image preprocessing path. Different dimensions change the grid. A scan with fine print can therefore create a tradeoff between retaining readable detail and conserving sequence space.

The image tokenizer is trained for reconstruction rather than a CLIP-style semantic matching objective. That distinction helps explain why visually representing a page and reliably interpreting all of its contents are separate achievements. A representation can retain local visual information without making every relationship equally easy for the language model to learn. Document understanding, object recognition, spatial reasoning and precise bounding-box prediction should consequently remain separate evaluation categories.

The audio path uses 24 kHz mono input and a single codebook with 4,096 entries. At 600 audio samples per code, it produces 40 codes per second. A minute therefore contributes approximately 2,400 audio codes before surrounding markers and instructions. This arithmetic provides a concrete planning estimate for this tokenizer. It does not measure transcription accuracy, preprocessing latency or end-to-end serving throughput.

Consider a hypothetical archive application accepting a page image and a one-minute spoken explanation. At the dimensions above, media alone accounts for about 3,424 codes. The written question, conversation history and output allowance add to that. If a system accepts twenty such bundles, the input is not twenty short messages; it is a substantial multimodal sequence. That changes admission control and the amount of memory the server must reserve for concurrent requests.

The more subtle issue is resolution allocation. A user might send a full page to ask about one small handwritten correction. Downsampling can erase exactly the evidence the question depends on, even when the page remains recognizable. Cropping could preserve that detail with fewer codes, but a crop can remove the heading that explains what a number means. These are application design choices to test against real documents, rather than capabilities established by a nominal context limit.

The vocabulary is an artifact worth testing

The report separates the original text vocabulary, reserved tokens, image codes and audio codes. Their total differs from the padded embedding dimension. Padding exists for implementation reasons; it does not add meaningful language or media symbols. Confusing these counts can lead an integration to accept an invalid token ID or to infer a capability from an unused embedding position.

The omni-tokenizer repository makes this otherwise obscure part of the release concrete. It provides canonical tokenizer artifacts, construction recipes and validation checks. Its documentation explains compatibility differences between tokenizer and framework versions, and warns that casually re-saving artifacts can change files or metadata. A successful load is therefore weaker evidence than a validated reproduction of the intended tokenization.

Imagine two servers using the same numerical weights but different chat templates. One inserts the intended end-of-turn marker; the other emits ordinary text that looks similar to a human. The requests are no longer equivalent at the model boundary. A disagreement in output cannot immediately be attributed to model randomness or hardware. The preprocessing pipeline has changed the experiment before inference begins.

Checksums address one part of this problem: whether a file matches a known artifact. They do not establish that a model answers questions correctly. A release can have perfectly consistent tokenizer files and still make factual errors. Keeping those forms of assurance separate is useful because they call for different remedies. Artifact inconsistency requires version control and validation; a task-quality failure requires evaluation, adaptation or a different model.

This is also why tokenizer maintenance belongs in an operational model inventory. Teams often record the checkpoint name and quantization but omit the processor, template and image preprocessing version. A multimodal release makes that omission costly. If a preprocessing update changes crop order or audio normalization, the deployed service can change while the weight checksum remains constant.

The two sizes do not share an identical journey

Apertus 1.5 continues from the earlier Apertus family rather than training both sizes from scratch. The report describes four trillion additional training tokens for the 8B run and two trillion for the 70B run. The model sizes also differ in how much multimodal material they consume. These histories matter when interpreting an unexpected result in which the smaller model performs better on a particular modality benchmark.

The authors report that the 70B model trails the 8B on several multimodal evaluations. They connect this to its incomplete consumption of the multimodal mixture and to an inherited learning rate that gave it less room to adapt. The first is a documented difference in the runs; the explanation about adaptation is the authors’ interpretation. Neither supports a general rule that smaller models are better at multimodality.

Post-training adds another distinction. The report traces supervised fine-tuning, long-context supervised training, reinforcement learning with verifiable rewards, direct preference optimization and self-distillation-based preference optimization. The final 70B lineage does not pass through every stage reached by 8B. Treating a family name as proof of identical procedures would erase a relevant experimental variable.

For a buyer, the practical consequence is to compare actual endpoints. Suppose an organization is choosing a checkpoint for multilingual document extraction. It should examine that task’s language, page layout and error costs at each model size. A more expensive endpoint may offer better general reasoning while a smaller endpoint performs well enough on the required document distribution. Parameter count is a useful resource descriptor, but it cannot replace that comparison.

For a researcher, the intermediate lineage is even more useful. If a behaviour improves after long-context training and regresses after preference alignment, the final model alone obscures the relationship. Releasing stages makes it possible to formulate a more precise question: which intervention changed this behaviour, and did it help other behaviours at the same time? That question is more actionable than asking whether the final model is simply good or bad.

Alignment can improve answers and loosen boxes

One of the report’s clearest findings concerns referring-expression grounding: identifying the image region described by a phrase. Preference alignment improves several multimodal categories but worsens this form of grounding by about seventeen points across the RefCOCO family. The authors’ diagnosis is that output formatting remains valid and the object is still located, while predicted boxes become systematically looser.

This is an instructive failure because a syntactic test could pass. A response containing four correctly formatted coordinates is not necessarily a precisely calibrated localization. If the downstream application needs to crop a label, isolate a defective component or select an object, extra background can change the result. Valid structure is only one layer of correctness.

The release accepts this tradeoff rather than claiming uniform improvement. That decision is valuable evidence about the limits of broad preference optimization. Human or model preferences for a helpful response need not reward the geometric exactness of a bounding box. An objective that improves a collection of conversational tasks can therefore move a specialist metric in the wrong direction without breaking the output schema.

A hypothetical quality-control interface illustrates the distinction. A broad box may be sufficient for a reviewer who merely needs to locate a damaged item in a photograph. The same box may be unsuitable for an automated measurement step that estimates its dimensions. These are different product requirements even when they share the same prompt. The proper acceptance threshold should follow the downstream use, rather than a broad claim that the model understands images.

The broader lesson is methodological. A model comparison should record not only successful answers but also whether the failure changes category. A malformed box, a precise box on the wrong object and a loose box around the right object require different fixes. The Apertus report makes that distinction explicit, giving downstream users a better basis for designing their own evaluations.

Thinking and tool use stop at a real boundary

Optional thinking is a request-time mode rather than a separate family of weights. That can be useful for workloads where additional generated reasoning improves an answer. It also consumes output budget, so the server must allow enough generation for the model to reach its visible response. A configuration that truncates the reasoning span can appear to produce no answer even though the model has been generating tokens normally.

But thinking should not be combined with tool use in this release. The technical report explains that the supervised mixture lacks conversations that both reason and call tools, and tool-use prompts are excluded from reinforcement learning. It also documents a serving-parser problem with calls emitted inside thinking. The model cards accordingly publish distinct recipes. This is a training and integration boundary, not merely an omitted example.

The difference between representability and learned behaviour is central here. A chat template can contain syntax for tool calls within a reasoning span. That establishes that a sequence can be encoded, not that the model has learned to choose, populate and recover from those calls. A language grammar can express a database transaction without giving its speaker the competence or permission to execute one.

The report additionally limits tool evaluation to single-turn categories and says the release was not designed for agentic use. That narrows what a developer can infer from a tool-call score. Multi-step workflows introduce observations, failures, retries and state changes. An initial correct call does not establish that the system will interpret its result correctly or stop before taking an unauthorized next action.

For a production integration, the immediate implication is to define separate modes deliberately. A document-answering endpoint can use thinking without exposing a tool interface. A tool endpoint can constrain available actions and validate arguments without implying the model performs an extended reasoning loop. Combining them should be treated as additional development and evaluation work, rather than an advertised capability hidden behind an undocumented flag.

The data inventory reveals work a download cannot perform

The multimodal data repository provides preparation scripts and a dataset inventory, including modalities, stages, upstream sources and licensing information. Its scope includes downloading, deduplication, conversion, recaptioning, pairing and tokenization. That sequence shows why a dataset name alone is not a reproducible training recipe: the transformations between the original material and the final tokens can matter as much as the source.

Availability also has several meanings. Some upstream collections require access approval, and weights used for recaptioning are not simply bundled with every preparation script. Public identification therefore does not mean every reader can instantly fetch every ingredient from a single unrestricted archive. Reproduction depends on obtaining the same inputs and understanding the access conditions attached to them.

The data transparency repository publishes versioned catalogue information. Its described contribution and legal-review workflow includes internal steps before public synchronization. That is a useful distinction: the public output of a governance process can be inspectable without every internal interaction being public. It would be inaccurate to transform a released inventory into a claim of complete visibility into all institutional decisions.

The legal repository collects policy and transparency documents. The report explicitly avoids asserting that publishing them proves legal sufficiency. Model weights and code licensing must also be separated from source-data licensing. An organization using the release should read the relevant terms for its intended activity, rather than assume one Apache label extends automatically to every upstream work.

These distinctions make openness a set of concrete questions. Can a team inspect what data was used? Can it identify a version? Can it rerun a transformation? Can it obtain the relevant source under its conditions? Can it redistribute the resulting artifact? Each question may have a different answer. The value of a detailed release is that those answers become researchable rather than disappearing behind a broad adjective.

Failures in the recipe are part of the result

The report describes data and infrastructure problems, including artifacts in recaptioning templates and a checkpoint conversion issue involving positional scaling. Those details can look less impressive than benchmark gains, yet they carry substantial practical information. A conversion can change apparent long-context performance without changing the intended training method. A template artifact can enter the data pipeline long before anyone notices it in model behaviour.

This expands the definition of reproducibility. Repeating an optimizer setting is insufficient if the input stream was altered by a preparation script or the exported checkpoint lost a configuration parameter. The reproducer needs the relationship between source data, transforms, training state and deployment artifact. A failure along any edge of that chain can create a model that has the right name and the wrong behaviour.

The post-training repository adds configurations and training infrastructure that can support this investigation. Its examples and supported recipes must still be matched to the specific release and stage being reproduced. A public codebase is not proof that someone has independently completed the entire run. Nor does a successful small experiment establish that the original large-scale workload has identical numerical or operational properties.

A useful original analogy is a scientific instrument calibration record. Publishing the final observation is valuable; recording a discovered calibration problem is also valuable because it tells future users what to verify. In model development, tokenizer hashes, conversion settings and data checks can play similar roles. They do not certify every conclusion, but they help prevent a known class of mistaken comparison.

This perspective also improves incident diagnosis after deployment. If a service suddenly loses document retrieval quality, a team should not immediately conclude that the model’s capability changed mysteriously. It can compare exported configuration, preprocessing and serving versions before retraining anything. The public recipe supplies candidate explanations and artifacts against which to test them.

A deployment manifest is the bridge between the artifacts

The released files answer different questions, so a deployment manifest should preserve their relationships. Recording just “Apertus 1.5” loses both size and stage. Recording a final checkpoint but omitting the tokenizer loses the mapping from user content to input IDs. Recording both but omitting the runtime branch loses parser and processor behaviour. These are concrete reasons to keep a small, versioned manifest beside the service configuration.

Take an audio archive as a concrete example. The ingest service could retain a source recording and preprocessing metadata separately from its transcript. If a later processor changes channel handling or sample conversion, the team could replay the same source through both versions. That comparison would help isolate a preprocessing difference before attributing a transcription change to the checkpoint. It would not by itself determine which transcript is correct; that still requires a reference or human review.

The report’s audio training distribution makes this especially relevant. Parliamentary and political speech is not interchangeable with every recording an archive might contain. A quiet prepared address, overlapping conversation and a noisy field interview differ in acoustics and structure. The release’s transcription evidence can inform a starting hypothesis, while a local evaluation must establish whether the intended material is represented adequately. This is a distribution question, rather than a problem solved by supporting an audio input field.

A similar distinction applies to a multilingual document service. It might process the same form in several languages, with the answer always located in a fixed box. A translation-quality evaluation would not completely test that system because the visual location and extraction format also matter. Conversely, accurate extraction from one language’s form would not establish that the model understands unfamiliar script or phrasing elsewhere. Keeping language, layout and output format as separate variables makes failures easier to diagnose.

The report also describes security filtering around large-scale data ingestion after requests to problematic destinations affected shared network infrastructure. The lesson is specific: preparing public web data is an active network workload, with redirects and destination checks, rather than merely reading an inert local archive. Someone reusing preparation code should understand that operational boundary. The publication of a source URL does not make every current destination safe or permanently equivalent to its earlier contents.

For future model releases, this combination of manifest, data lineage and disclosed failure is a useful standard to ask for. It enables a team to state what changed between two deployments and what stayed constant in an experiment. That is stronger than a promise that everything is reproducible, because it identifies the artifacts against which such a promise can actually be tested.

What remains uncertain after a transparent release

The report is candid about factual reliability, uneven language coverage and limitations outside speech transcription. Stronger long-form answers do not establish reliable short factual recall. Training on many languages does not mean each has equally thorough instruction or reasoning evaluation. A system used for a particular language or specialist domain still needs evidence from that distribution.

Its long context also needs careful interpretation. Capacity means the model can accept a sequence within an architectural and serving limit; it does not guarantee faithful retrieval from every position or robust synthesis across contradictory documents. The report follows long-context behaviour through training stages, including a regression after shorter supervised work and recovery with long-context training. That history demonstrates why context quality should be measured rather than inferred from a maximum length.

For an organization deciding whether to deploy Apertus, three different kinds of confidence must remain separate. Artifact confidence concerns using the intended files and preprocessing. Task confidence concerns measured performance on relevant examples. Operational confidence concerns memory, concurrency, latency and failure recovery in the chosen runtime. The release improves the evidence available for each, but none follows automatically from the others.

An evaluation could begin with fixed document sets whose answers and evidence locations are known, then vary page resolution, language and context length independently. It could separately record unsupported answers, missing evidence and localization errors. This is a proposed evaluation design, not a test we ran. Its purpose is to make deployment questions specific enough that an open recipe can help answer them.

Apertus 1.5 therefore offers something more durable than a ranking claim: a model family whose tradeoffs can be traced. The expanded tokenizer explains its multimodal interface, the checkpoint lineage explains some size differences, and the limitation section blocks several tempting overclaims. For readers comparing releases, our guides to model cards, self-hosted models and reasoning models provide complementary context. The decisive question is which published artifact answers the actual deployment question, and which answer still requires a new experiment.