GPT-6.1 Terra and the shift from reading screens to acting on them

An assistant that can read a screen can explain what is visible. An assistant that can work through a screen must connect that interpretation to an action and its consequences. GPT-6.1 Terra’s positioning makes this distinction timely: useful computer work depends on several capabilities that a convincing description alone cannot establish.

OpenAI’s model documentation describes Terra as a model for visual reasoning, computer use and writing. Its API changelog records the release on 26 October. The model accepts text and image input, produces text, and requires the Responses API for tool calling. Those are documented interfaces and intended uses, rather than findings from a test by this publication.

The practical change is a longer chain of responsibility. A visual answer ends with an interpretation. An application task can leave a changed file, record or message outside the conversation. This analysis separates four task types: perception, action, state verification and recovery. The taxonomy and worked scenarios are our own. We have not run Terra against the applications described here.

Perception establishes what the screen means

Perception is the task of connecting visible information with the user’s question. In an invented warehouse example, a screenshot contains product names, stock counts and a filter. The assistant must determine which number belongs to the requested product and whether the filter changes the population being shown. Reading all the words correctly would still be insufficient if it assigned a count to the wrong row.

This is a useful distinction for visual reasoning because mistakes can occur before any action is proposed. The system might miss a status icon, confuse similar names or overlook that the page is showing archived records. An accurate general description could coexist with a wrong answer to the specific question. The relevant unit of understanding is the relationship the task requires.

A developer can make that relationship explicit. For the warehouse question, the expected answer might identify the product, the displayed count and the active filter. These fields create a checkable interpretation rather than an impression that the screenshot was understood. They also make a later failure easier to locate: the model selected the wrong record before it ever touched a control.

OpenAI’s vision documentation identifies limitations involving small text, rotation, counting and spatial localisation. Those documented categories support inspecting the actual image supplied to the system. They do not supply a Terra error rate for this warehouse task. The application needs its own evidence about the material it expects to encounter.

For a read-only assistant, a correct interpretation may be the entire useful result. There is no reason to require application actions simply because a model can request them. A well-bounded visual question can deliver value with a smaller execution surface. The task should decide which additional capabilities are necessary.

Action connects a decision to an external operation

Action begins when the system tries to change something outside its own answer. The operation might select a filter, enter text or invoke an application function. The key distinction is that the request now has an intended consequence in an external system. Saying what should happen and making it happen are different events.

The function-calling guide describes an application-side execution loop: the model requests a tool, the application executes the relevant operation and the result returns to the model. This separation matters even when an interface makes the whole exchange feel like one assistant action. The generated request and the actual operation have different failure modes.

Consider an invented instruction to mark a shipment for collection. The assistant can correctly understand the request while passing the wrong record identifier to a tool. It can pass the right identifier while the tool lacks permission to change the record. The tool can accept the request while the application queues work that has not completed yet. Each case needs a different explanation.

An action record should therefore describe the intended object and operation. “Clicked the blue button” is a weak account if several buttons look alike or the interface changes. “Requested collection for this shipment” is closer to the user’s task, but still needs evidence connecting that intention to the object actually changed. Semantic intent and mechanical delivery must meet at the same target.

This does not require exposing implementation details to every user. A well-designed product can present a concise confirmation while retaining the underlying record needed for diagnosis. The important product requirement is that the confirmation represents the operation that occurred, rather than a fluent prediction of what normally follows a click.

Verification identifies the state that actually matters

State verification asks whether the task’s intended outcome now exists. This is distinct from interpreting the starting screen and from delivering an action. The evidence should match the user’s goal. A changed document, an updated field and a completed export are different kinds of outcome, even if all involved pressing a button labelled Save.

In an invented chart-editing task, the user wants a presentation to show a corrected unit. The assistant changes the visible label and announces completion. If it edited an unsaved preview or the wrong slide, the visible action was real but the requested outcome remains absent. The meaningful check concerns the deliverable and location the user intended to change.

The same reasoning applies to an online form. A visible success notice may be strong evidence for a routine submission. A validation error is evidence that the task remains unfinished. A disappearing dialog can be ambiguous. The system should identify the application’s strongest available signal rather than treating any screen transition as success.

Independent outcome checks have a research history. The OSWorld paper introduces computer tasks with initial-state configurations and task-specific execution-based evaluation. Its relevance here is methodological: a computer-use result can be judged against the resulting environment. Its historical model scores do not establish Terra’s current performance and are not used as a comparison in this article.

A useful completion report follows the same discipline in ordinary language. It states the outcome established and any remaining uncertainty. If the application has confirmed that an export is queued, the assistant should say queued. Calling it delivered would skip a stage for which no evidence has yet appeared.

Recovery is a task with a different starting point

Recovery begins after the system’s expectation and the observed environment diverge. A dialog appears, a session expires or the application rejects a value. The assistant now needs to identify what is still true before deciding what to do next. Repeating the previous action can be the wrong response even when the original action was sensible.

Imagine a fictional task that adds a row to a shared list. The application times out after the request is sent. There are at least two possibilities: the write failed, or the write succeeded and the confirmation failed to return. Blindly repeating the operation could create a duplicate. Reporting failure without inspection could also be wrong. The immediate problem is uncertainty about state.

A recovery procedure should seek evidence that distinguishes those possibilities. The application might offer a stable identifier or a view of the resulting list. If it provides neither, the assistant may need to report that it cannot establish the outcome. That is a useful limit, because it prevents an uncertain operation from becoming an unexamined sequence of repeated changes.

Recovery also needs to preserve the user’s original scope. An expired session does not authorise creating a new account. A missing record does not authorise substituting a similarly named one. The task remains the same task unless the user changes it. A competent recovery process restores the relevant conditions or exposes the obstacle.

For product designers, this makes recovery a separate capability worth exercising. A demonstration that begins in an ideal state cannot reveal how the system handles interruption. The useful question is whether it can resume from the states the intended application actually produces, with a clear account of what was already done.

Observation and action have a shared reference problem

An action can be correct relative to an observation that is no longer current. This problem is especially visible with screenshots. The image shows a control in one location, but the page can move, expand or replace that control before input arrives. The system needs a relationship between the observed target and the current target.

A coordinate is a position in a particular image and display arrangement. It is not an enduring identity for a record. A browser interface that exposes labelled elements can provide a different reference, but that reference also needs a defined lifetime. Neither approach eliminates the need to ground an operation in the current application state.

OpenAI’s driver documentation explicitly distinguishes fixed approved arguments from a screen that remains free to change. Approval does not freeze the document. The application’s execution path must preserve target validity when the permission depends on a particular object. That is a concrete integration obligation, rather than a model-quality score.

In our fictional shipment workflow, the important identity is the shipment the user selected. If a sorted table refreshes, a row position may refer to another shipment. A useful design carries the intended identity through the operation or refuses to act when it cannot establish that relationship. Re-reading an unstable page without retaining the intended object does not solve the problem.

This explains why a computer-use trace needs more than attractive screenshots. It should connect observations, target references and actions in order. Without that connection, a reviewer can see a sensible plan and a wrong result while remaining unable to identify whether perception, target selection or execution caused the mismatch.

The environment determines what the model can do

A model’s tool support does not automatically supply a logged-in browser, access to a company’s records or a running desktop. The application provides those resources. It also decides which operations are exposed and how long the session persists. Model availability and a usable task environment are separate parts of a working product.

OpenAI’s computer-use guide describes connecting model interaction with browser or desktop execution. That documented arrangement supports a useful mental model: the model proposes or selects operations within a system that owns the actual environment. The system needs to return observations of what happened so the next decision reflects reality.

Consider two fictional assistants using the same model. One receives a screenshot and a broad mouse-and-keyboard tool. The other receives structured page information and a narrow function for updating one approved field. They have different observations and action surfaces. If their outcomes differ, the model identifier alone cannot explain the result.

The distinction matters when comparing products. A complete product can legitimately be evaluated with all of its integration advantages. A claim about the underlying model requires a more controlled comparison. Otherwise, a better driver, a cleaner initial session or a narrower task can be mistaken for a general model advantage.

Our agent-harness explainer develops this system-level view. The harness manages the loop around a model, including tools and retained state. Terra adds another documented model option to that loop. Whether the full system is useful depends on how those parts support the particular work being delegated.

Data on a screen does not grant new authority

Computer work often involves reading material the user did not author. A page, attachment or receipt can contain instructions as well as task data. Treating all visible text as equally authoritative would let the material being processed redefine what the assistant is allowed to do. The observation channel and the instruction channel need different roles.

An invented support ticket might include a request to send unrelated customer records to another address. The ticket is relevant evidence for the support task, but its presence in the interface does not establish that the user authorised that disclosure. The system needs to retain the actual task and permission boundary while interpreting the document.

This is the connection to prompt injection. A model can read a malicious instruction accurately and still need to disregard it as authority. Better visual recognition alone does not answer the trust question. The issue is what status the system assigns to the recognised content and which actions the execution path permits.

For a proposed application, a narrow tool can make the boundary easier to express. A tool that changes one approved field has a different consequence surface from unrestricted desktop access. This is an architectural observation, not a measured claim that one integration makes Terra immune to injection. Any specific assurance would need evidence about the actual controls.

The product should also make a legitimate boundary visible to the user. If the assistant cannot complete a task because the required destination is outside its allowed environment, say so. A transparent limitation is more useful than either a silent expansion of access or a vague failure that hides the missing capability.

A task can change category as it unfolds

The four task types are analytical categories rather than a mandatory sequence of four model calls. One operation can perform several roles, and a workflow can return to an earlier role. Reading a confirmation can both interpret a new observation and verify an action. A recovery step can require further perception before it can choose a safe operation.

That flexibility is important for avoiding a mechanically inflated agent. A simple question may need one observation and one answer. A complex application change may require several cycles. The objective is to obtain adequate evidence for the task, rather than to maximise the number of intermediate steps or force every request through the same elaborate routine.

Imagine an invented invoice task that begins as extraction. The assistant reads the total and supplies it to the user. If the user then asks it to enter that total into a different application, the task gains an action component. If the application rejects the format, recovery becomes relevant. The original extraction accuracy remains necessary throughout, but it no longer describes the whole job.

A product team can use the taxonomy to locate its current scope. Is the product promising an answer, a proposed change or a completed operation? Does its interface clearly distinguish these states? Ambiguous language can cause users to rely on a draft as if it were a committed result, even when the underlying components behaved exactly as designed.

This also makes incremental deployment more concrete. A team can begin with read-only assistance, assess its interpretation failures and then decide whether to expose bounded operations. The transition should follow evidence about the workflow. Adding a computer tool because it exists in a catalogue is not itself a reason to expand the product’s promise.

Measurement should preserve the different failure types

A single completion rate can be useful, but it cannot explain why a workflow succeeds or fails. A perception error, an unavailable operation, a rejected submission and an unresolved timeout all leave work unfinished for different reasons. A report that preserves those categories gives developers a more useful basis for improvement.

OpenAI’s evaluation guidance recommends task-specific objectives, datasets and metrics. We use that principle here as guidance for designing a study, without recommending the availability of any particular hosted evaluation product. The intended output is a measurement plan matched to the application, rather than a new Terra score.

A proposed evaluation can score the selected object, the requested operation and the final state separately. It should retain cases in which the system appropriately stops because the target cannot be established. A process that refuses an ambiguous update may be behaving correctly, even though it does not produce the external change requested in the original task.

Historical agent research also makes clear why an interactive setting matters. AgentBench evaluates decision-making across multiple environments. Its relevance is that multi-step interaction creates demands beyond a standalone answer. Its results concern the evaluated systems and settings, so they should not be presented as evidence that Terra has solved those demands.

A report should describe the initial state and any human intervention. An assistant that needs someone to select the correct record has received substantive help, even if it performs every subsequent click. That help may be entirely appropriate in a product, but it belongs in the account of what the system accomplished.

Recovery changes the resource calculation

The amount of work in an agent task is partly determined by how often the system must revisit its assumptions. A concise successful path and a long sequence of failed retries can begin with the same user instruction. The difference affects elapsed time, model usage and the human effort required to resolve uncertainty.

These costs should be connected to completed outcomes. A demonstration with no interruption can show what an efficient path looks like, but it cannot establish the average cost of a workflow that regularly encounters changing state. A useful assessment includes the relevant distribution of starting conditions and records unsuccessful attempts as well as successes.

There is also a design trade-off between observation and action. Inspecting too little can cause avoidable mistakes; inspecting repeatedly without a concrete uncertainty can waste time. The right observation is one that answers a decision-relevant question. Is the target still the intended object? Did the operation succeed? What changed after the error?

For the fictional shipment task, an authoritative result identifying the updated record may settle the question without further screenshots. An ambiguous timeout may justify another read. The required effort follows the evidence gap. This is a proposed efficiency principle, not a claim that Terra implements a particular observation policy by default.

A useful product review would therefore publish both the outcome and the path needed to establish it. That gives readers a way to assess whether an apparently successful assistant saves work in their own environment. A bare token rate or an impressive uninterrupted video cannot provide the same information.

A correct display can conceal an incorrect artefact

State verification sometimes needs to go deeper than the screen that announced success. Consider an invented spreadsheet task in which the user asks for a corrected calculation. The assistant changes a displayed total to the expected number. The screenshot now looks right, but a constant has replaced a formula. When the inputs change, the sheet no longer performs the requested calculation. The visual result and the functional result have diverged.

The appropriate check follows the deliverable’s semantics. If the task was to repair a formula, inspect the formula and its response to relevant inputs. If the task was to export a particular table, inspect the exported rows rather than only the export dialog. These are analytical examples, not observations of a Terra failure. They illustrate why the user’s verb matters: repair, calculate, display and export name different obligations.

This creates a useful distinction between surface verification and functional verification. Surface verification can establish the presence of the intended label or value. Functional verification examines whether the object still behaves as required. A task may need one or both. The evaluator should not silently substitute the easier check when the user’s request depends on the harder property.

The same principle helps scope human review. An expert need not watch every click to inspect a final formula, and a screenshot reviewer may be unable to establish its behaviour. Assign the review to the evidence required, rather than to whichever trace is most convenient to display. A concise functional check can sometimes be stronger than a long visual recording.

The observation can omit the answer

A visual assistant can only interpret the material it receives. In a fictional catalogue task, the relevant entry may be outside the current viewport or hidden behind an active filter. A confident answer based on the visible subset would answer a narrower question than the user asked. The first reasoning problem is whether the supplied observation can support the requested conclusion.

The system should distinguish absence from the current view from absence from the application. Searching another page or adjusting an authorised filter may be necessary. When the available tools cannot reveal the missing information, the appropriate response is to identify the gap. Guessing from a partial screen does not turn that partial screen into complete evidence.

This also affects evaluation design. A test that always places the answer in the initial image examines a different task from one that requires locating it. Both can be useful, but they should have separate descriptions. Keeping the distinction visible helps a builder identify whether the next improvement concerns interpretation of available information or acquisition of information the model has not yet seen.

The product promise should name its stopping point

The clearest way to assess a screen-based assistant is to ask what it promises to leave behind. A description, a proposed edit and an updated application are different deliverables. A product can be valuable at any of these stopping points when its interface and evidence match the promise. Confusion arises when it presents one as another.

For Terra, the documented release provides a concrete starting point for that assessment: a model positioned for visual and application work through specified interfaces. It does not eliminate the need to choose an environment, define permitted operations or check the result. Those decisions are part of the product that a user actually encounters.

A builder should begin with one meaningful task and identify the evidence at each boundary. What must the system recognise? Which operation may it request? Which application signal establishes the outcome? What should happen if that signal never arrives? Answering these questions creates a useful specification before a model comparison begins.

The shift from reading to acting is therefore a shift in what counts as a finished answer. Fluent text can explain the work. A completed task requires the relevant state of the world to agree with that explanation. The strongest screen-based assistant makes that agreement inspectable, and says clearly when it has not yet been established.