Can one small model inherit several specialists' skills?

A specialised language model can become good at one job while remaining a poor choice for another. The harder engineering problem is to carry several specialists’ strengths into one deployable model without allowing the easiest capability to dominate the training signal.

P²-MOPD, a 2 November preprint, investigates teacher feedback combined with learned parameter priors in 1.7B and 4B models across four domains. The paper provides a concrete case for examining how a compact generalist can acquire specialist capabilities.

The broader significance is the distinction between knowing where to start and knowing how to continue learning. A promising merged checkpoint does not guarantee that later training will preserve every useful capability. For teams considering a compact specialist, the paper offers a reason to inspect the path of consolidation and the weakest important task, rather than only the final average.

A deployable generalist is more than an average specialist

Imagine a hypothetical internal assistant that explains technical procedures, follows a strict output format and answers questions about financial documents. The organisation already has models adapted to different parts of that work. Keeping all of them in service could preserve specialisation, but it creates routing, operational and evaluation requirements. Training one consolidated model might simplify deployment, if the resulting behaviour is good enough.

That final condition matters. The company does not benefit from a model that becomes excellent at formatting while losing an important numerical capability. Nor does a strong average across convenient tests establish that the assistant is reliable on the unusual questions employees actually ask. Consolidation is a constrained engineering problem, not simply an effort to maximise one score.

The distinction has a long history. Hinton, Vinyals and Dean’s distillation paper describes transferring knowledge from an ensemble into a single model that is easier to deploy. Its experiments and specialist construction are historical context, not evidence for today’s language-model configurations. The enduring idea is that the model used to provide training guidance need not be the model served to users.

This separation can make the economics attractive. Expensive or numerous teachers might be used during development while a smaller student handles later inference. But a deployment saving is a hypothesis until training cost, maintenance and actual task performance are counted. A cheaper response that requires more human correction can erase an apparent infrastructure gain.

For this reason, an evaluation should begin with the intended work. Specify the tasks, the acceptable errors and the conditions under which the assistant must defer. Then assess whether consolidation meets those requirements. Starting from an attractive model size and searching for a benchmark that justifies it reverses the decision.

Our guide to self-hosted models covers the serving side of that calculation. The training question is complementary: which capabilities survive in the checkpoint you intend to run? Hardware compatibility cannot answer it, and a successful load test cannot reveal whether knowledge from one teacher displaced another.

Teachers can guide outputs and parameters differently

Behavioural distillation and parameter combination operate at different boundaries. A teacher can score or demonstrate an output. A weight-space operation can incorporate information contained in compatible model parameters. The first concerns what a student should do in a context; the second changes the numerical object that performs the computation.

A simple educational analogy helps. One instructor can mark a student’s attempted solution and explain where it diverges. Another resource might provide an initial problem-solving routine. Having a useful routine does not remove the need for feedback, while receiving feedback does not guarantee that the student can efficiently discover every useful routine from scratch. The analogy describes different kinds of guidance, not a biological account of neural networks.

The MOPD paper by Ma and colleagues describes domain teachers supervising a student’s own rollouts. Its reported model and industrial applications are evidence for that work, and should not be silently reassigned to a later method. The relevant distinction is that the student generates the trajectories on which the routed teacher supplies guidance.

Thinking Machines Lab’s account of on-policy distillation explains why this can differ from learning only from teacher-generated sequences. A student can enter contexts the teacher seldom visits, particularly after an early mistake. Feedback on the student’s own continuation addresses those visited contexts rather than assuming the student always remains on a teacher’s preferred path.

That idea has practical implications for task design. If the assistant frequently makes an incorrect intermediate assumption, training only on pristine completed answers might leave the recovery behaviour unexamined. A teacher evaluating the student’s attempt has a chance to supply a more relevant signal. Whether the signal is good still depends on the teacher, objective and data.

Parameter arithmetic introduces a different opportunity. Editing Models with Task Arithmetic constructs task vectors from the difference between a fine-tuned model and its base, then explores arithmetic operations on those vectors. The operation is not a request to several independent APIs. It requires a meaningful relationship among the parameters being combined.

The distinction prevents a common category error. A company may possess strong models from several unrelated sources without possessing compatible checkpoints for direct weight arithmetic. Distillation could still be considered at the behaviour boundary, subject to access and licence constraints. Directly adding tensors because both models answer similar questions is not an equivalent procedure.

The table separates mechanisms. It does not rank them universally or imply that every row is needed in every training system. A clear experiment states which ingredients changed and holds the other conditions steady enough for the comparison to mean something.

A useful prior is not a permanent guarantee

In the paper’s 1.7B experiment, average recovery reaches 94.7%, against 80.9% for a control with the same merged starting point and frozen coefficients. These are domain-normalised recovery figures across the selected experimental tasks, not universal task accuracies.

Those details make the experiment more informative than an unmatched comparison between a carefully initialised method and a weak starting point. They also limit the claim. A result at these scales does not establish that the same gains occur in larger architectures, unrelated checkpoints or operational tasks absent from the evaluation.

The general optimisation question is easy to state. A starting checkpoint can already contain useful behaviour, yet subsequent updates can move it away from that behaviour. If a training objective rewards some tasks more effectively than others, the final model may become unbalanced. Measuring only the start and a final average can conceal when and where the imbalance emerged.

An experiment designed around this question should inspect intermediate checkpoints. It should track each domain, not merely an aggregate loss. It should compare methods from a matched initial state when the claim concerns the update procedure. It should also identify whether a model selected for presentation was chosen using validation feedback unavailable to the baseline.

This is methodological analysis, not a report that we retrained the paper’s models. It explains why the shape of the evidence matters. A convincing mechanism study does more than announce a stronger checkpoint: it changes an ingredient, observes the consequence and tests plausible alternative explanations.

One alternative explanation is that a method received a better initial mixture. Another is that it used more compute. A third is that a difficult domain received more effective training exposure. These explanations can coexist with a useful method, but they change the conclusion. The correct claim might be that an entire recipe works better, rather than that one isolated mathematical component caused every gain.

For an applied team, the distinction affects reproduction. If the benefit depends on a particular start, teacher bank and training mix, reproducing only the headline update rule may disappoint. A deployment plan should list those dependencies explicitly and establish which of them the team can actually reproduce.

A two-coordinate example makes the limitation visible

Consider a deliberately simplified mathematical example with only two adjustable coordinates. Let a base sit at the point zero, and let the available expert direction move only horizontally. Changing the coefficient on that direction can move a merged candidate left or right. It cannot move the candidate vertically. No choice of that coefficient reaches a solution above the horizontal line.

Now allow the student’s coordinates themselves to change. The candidate can move vertically as well as adjust its contribution from the expert direction. The example does not simulate a language model or establish a performance result. It explains the geometric difference between selecting a combination from a fixed set of directions and adapting the underlying model more freely.

Add a second expert direction pointing diagonally. The available combinations can now reach a broader part of this toy plane, but the practical question remains: which movement improves the actual objective? A useful direction is a resource, not an instruction to move as far as possible. Large contributions can overshoot or conflict with other requirements. An optimisation procedure needs feedback about the combined result.

The analogy also explains why the number of coefficients is not a measure of total trainable capacity. A system can have relatively few mixing coefficients while maintaining a much larger set of trainable student weights. Counting only the small coefficient table would misdescribe what is being optimised. Conversely, freezing the student and learning only a mixture imposes a different constraint.

For an engineer reading a method description, the practical task is to identify what is frozen, what receives gradients and what is exported. A teacher can remain frozen while its derived direction participates in a trainable combination. A compact deployment artefact can arise from a training system that temporarily holds several models. Serving complexity and training complexity should therefore be reported separately.

This toy picture has limits. Real neural-network behaviour is nonlinear, a parameter direction does not correspond neatly to one human skill, and an objective can reward an imperfect proxy for the application. The picture is useful for understanding freedom of movement, not for predicting that a particular coefficient change will improve a medical or financial answer. That prediction needs experiment.

Interference has several forms, so the remedies differ

Model merging is not a single operation with one known failure mode. Different methods make different assumptions about which parameter changes contain useful information and how changes interact. Reading the earlier work helps prevent the new paper from being presented as if no one had considered conflicting specialists before.

TIES-Merging focuses on redundant parameter changes and disagreement over signs, using trimming, sign election and a merge restricted to aligned values. That is a specific approach to interference in its evaluated settings. It should not be reduced to the generic statement that “averaging models works”, because the intervention is precisely about why naive combination can lose information.

DARE, described in Language Models are Super Mario, instead explores dropping and rescaling delta parameters before combination. Its reported sparsity observations concern the models and experiments in that work. The provocative title is not a licence to call all model consolidation a free lunch. The practical method and the headline metaphor deserve different treatment.

AdaMerging learns merging coefficients using an unsupervised objective on unlabelled samples. It supplies historical context for learning how much of each model to incorporate. Its multi-task setting and objective differ from on-policy language-model distillation, so a reader should not assume the results form a common leaderboard.

LoRA is another useful boundary marker. It freezes a base and trains low-rank adaptation matrices to reduce the trainable parameter burden. Adaptation, merging and distillation can appear in the same broader workflow, but they are different decisions. A claim about parameter-efficient adaptation does not by itself establish successful consolidation of several specialists.

The editorial conclusion is that method names should follow the actual intervention. Did the team change the supervision, the parameter representation, the search space, the training mixture or the optimiser? A system can change several at once, but a comparison then needs to be clear about what it demonstrates. Treating every method as “combining models” is too coarse to guide engineering.

The hypothetical internal assistant illustrates why. A failed numerical answer could come from an inadequate teacher, insufficient numerical prompts, a destructive merge or a task outside the training distribution. Adding another teacher will not necessarily fix the right problem. A useful investigation should inspect failures and training records before selecting the next technique.

Read recovery as a gap, not an accuracy score

A relative progress measure can make tasks with different raw score ranges easier to compare. It can also be misread. P²-MOPD defines recovery relative to the gap between the base and the relevant teacher, then aggregates domains. A reported recovery percentage is therefore not the percentage of test questions answered correctly. Raw scores remain necessary to interpret the result.

Consider an explicitly hypothetical task where a base scores forty and a teacher scores sixty on a hundred-point scale. A student scoring fifty closes half the gap. Its recovery is fifty percent, while its raw score is fifty. The matching numerals here are an accident of the chosen example, not an identity between the metrics.

Change the hypothetical base to eighty and the teacher to ninety. A student scoring eighty-five again closes half the gap, now with a much higher raw score. The same recovery value describes different operational starting points. Conversely, a small raw improvement can represent a large fraction of a narrow teacher gap.

These examples explain why a product team should retain both views. Recovery asks how effectively consolidation preserves the teacher’s advantage over a chosen base. Raw performance asks how much of the evaluated task the resulting model can handle. Neither answers whether the task itself represents the intended application.

There are also edge cases. If the teacher barely improves on the base, a gap-normalised measure can become sensitive to small score changes. If the student surpasses the teacher, a relative recovery can exceed the intuitive ceiling suggested by the word “recovery”. An evaluator should explain the formula and handle such cases consistently rather than clip inconvenient results without disclosure.

Averaging adds another choice. Giving each domain equal weight says that each normalised gap matters equally in the summary. An application may have a different priority structure. A rare but consequential task could deserve a strict threshold, while a frequent low-stakes task could tolerate more variation. The publication should not turn a research aggregation rule into an unstated business objective.

A good presentation would therefore place the aggregate alongside per-task raw scores, uncertainty and the relevant reference points. It would state how outputs are sampled and scored. If several samples contribute to a score, that should remain visible when someone compares it with a single-attempt result from another system.

A proposed pilot for a compact internal assistant

For a team with compatible specialist checkpoints and permission to use them, the next step should be a controlled pilot. The following design is our proposal, not a procedure we executed. It begins with an application contract: what the assistant must answer, which format it must obey and when it must decline or escalate.

Build an evaluation set with separate categories. Include ordinary examples, ambiguous questions, multi-skill requests and deliberate conflicts between a correct answer and a requested output format. Preserve the provenance of each item so that training contamination can be investigated. A single blended test set makes it harder to distinguish broad progress from one category hiding another.

Keep a base, each specialist and a straightforward consolidated baseline. Record checkpoint lineage and the data used to produce the teachers. The purpose is not to make the experiment elaborate for its own sake. It is to establish what useful behaviour exists before consolidation and which component the proposed method is expected to preserve.

Then compare under an explicitly chosen budget. A fixed number of steps answers a different question from a fixed wall-clock allowance or a fixed accelerator budget. Reporting all three is useful when possible. If one method performs more work per step, a claim of needing fewer steps should not be translated automatically into the same proportional cost saving.

Track the worst important task during training. Set a rule for what counts as unacceptable regression before selecting the winning checkpoint. This prevents an attractive aggregate from rationalising a failure the application cannot tolerate. The rule should reflect the actual use case and be applied to every candidate consistently.

Evaluate with the intended prompt and decoding configuration. A compact model may look different when required to follow a strict response schema or answer within a limited token budget. If the application will use retrieval, include the actual retrieval conditions in a separate evaluation. The resulting system performance should not be attributed solely to training when other components also changed.

Finally, inspect a sample of errors with domain experts. A numerical score can reveal movement while concealing what kind of mistakes remain. Did the model lose a specialist fact, misunderstand a question, violate a format or become overconfident? The answer determines whether another training run, a tool or a narrower deployment boundary is the appropriate response.

Teacher quality is part of the experiment

A teacher’s expertise should be established on the task for which it supplies guidance. A model that performs well on a broad public benchmark can still be an unsuitable supervisor for a company’s terminology or document structure. The pilot should measure each teacher on the held-out application tasks rather than assume its label establishes domain authority.

The teachers may also disagree. A useful training design needs a rule for which signal applies to a mixed-domain prompt. That rule can be simple if the dataset has clear labels, but real user requests often cross categories. Evaluating only neatly separated tasks can leave an important deployment boundary untested.

This is one reason not to treat consolidation as merely copying expertise into a smaller container. The organisation must define what combined expertise means when goals conflict. A legally careful answer, a financially concise answer and a strict machine-readable answer can impose different constraints. The final application needs an explicit priority structure rather than hoping that parameter combination discovers the intended policy.

Consolidation changes the update process too

The pilot should also rehearse a later specialist update. Suppose the organisation improves its financial teacher after discovering a recurring document error. It now needs to know whether incorporating that change requires rebuilding every consolidated checkpoint, which old evaluations must be repeated and whether unrelated capabilities move. A one-time successful model does not answer the maintenance question.

Record the lineage of the updated teacher and keep the earlier consolidated model available for comparison. Test both the intended improvement and the critical tasks that should remain stable. A release note should state which training ingredients changed, rather than merely calling the checkpoint a newer version. That record makes later regressions easier to investigate.

There is a genuine operational trade here. One consolidated endpoint can simplify serving while moving complexity into training and release management. Separate specialists can be harder to route while allowing narrower updates. The right design depends on how frequently domains change and how expensive broad regression testing becomes. Consolidation is useful when its overall maintenance burden fits the organisation, rather than because fewer endpoints always imply a simpler system.

A release decision should also distinguish repeated sampling from independent training runs. Asking one checkpoint to answer the same item several times measures a different source of variation from training the method again with a different random seed. Combining both into a larger-looking sample count would overstate the amount of independent evidence. Keep the hierarchy visible in reports and confidence estimates.

For the proposed internal-assistant pilot, preserve item-level results and training-run identifiers. Reviewers can then examine whether an apparent gain comes from broad improvements or a handful of unstable questions. If an important category contains very few examples, report that limitation and expand the set before treating its score as a reliable deployment threshold. The cost of writing better evaluation items can be smaller than the cost of another training run that answers the wrong question. This is an evaluation recommendation, not an additional result claimed by the preprint.

The strongest conclusion is narrower and more useful

The study is interesting because it treats specialist weights as continuing guidance, rather than only a static initial mixture. The broader engineering implication is that a good consolidation recipe must account for how capabilities evolve during training. That question remains useful whether a later experiment confirms, narrows or improves on this particular method.

The immediate buying implication is more restrained. A company should not choose a small model because a recovery percentage resembles an accuracy target. It should choose a checkpoint only after measuring the tasks it will serve, the regressions it cannot tolerate and the full cost of maintaining that model. Our model-card field guide provides a companion checklist for interpreting the resulting evidence.

Nor should the availability of several expert models be mistaken for permission or technical compatibility to combine them. Check licences, lineage, architecture and data rights before planning a training run. Those are practical prerequisites, not details to resolve after a promising demonstration.

For researchers, the next persuasive evidence would test whether the mechanism persists across broader scales, task mixtures and independently reproduced pipelines. For builders, the next persuasive evidence is closer to home: a compact checkpoint that preserves every critical capability under the conditions in which people will use it.

Set that contract first. Report raw performance alongside relative recovery, compare honest budgets and keep the weakest required task visible. A generalist earns deployment by retaining the expertise its users need, rather than by inheriting an impressive average.