Repository navigation
GPT-5.6 Sol: acknowledged constraints are not reliably bound to subsequent actions #41851
Description
Full English evidence package
https://github.com/Ysaak62/chatgpt-technical-audit/blob/main/CODEX_READ_FIRST_EN.md
The linked entry file leads to the complete v1.3 audit in eight ordered parts and includes integrity hashes. The audit distinguishes observed facts, engineering interpretations, and internal causes that remain unknown without server-side telemetry.
What issue are you seeing?
I am reporting a repeated GPT-5.6 Sol reliability failure in which explicit user constraints are correctly acknowledged in text but are not reliably enforced in the model's subsequent action.
The primary evidence comes from a long-running ChatGPT multimodal/image-generation workflow. I am submitting it here because closely related GPT-5.6 Sol instruction-following failures are already being tracked in openai/codex, including #40033, #40424, and #41222.
The recurring pattern is:
explicit constraint → model acknowledges constraint → next action violates the same constraint → user corrects the model → model acknowledges the correction → later action may violate it again
Observed failure classes include:
- failure to preserve the approved source/reference set;
- execution inconsistent with constraints that had just been positively confirmed;
- failure of a prohibition to create an effective pre-action STOP;
- recurrence of an incorrect output family after direct corrective feedback;
- intermittent success under substantially unchanged task conditions.
A separate image-generation symptom was that the system could correctly understand the supplied images, while the subsequent generation behaved as though the intended references had not been preserved.
The audit does not claim an unverified server-side mechanism. The documented client-visible behavior is consistent with an instruction-to-action / state-to-action reliability failure. Determining the exact internal cause would require OpenAI telemetry.
What steps can reproduce the bug?
A controlled repeated-run test was started with the same task definition and the same intended source images.
- Establish a fixed source/reference set and explicit task constraints.
- Have GPT-5.6 Sol acknowledge those constraints before execution.
- Repeat substantially the same generation task multiple times without intentionally changing the required source set or target task.
- Compare whether the next action/output remains bound to the acknowledged state.
- After a clearly incorrect run, explicitly correct the model and instruct it not to repeat the incorrect behavior.
- Run the task again and observe whether the corrective instruction is reliably applied to the next action.
At audit version v1.3, 6 of the planned 10 runs had been completed:
- run 1: expected/acceptable output class;
- runs 2–3: radically different incorrect output family;
- run 4: returned to the expected class;
- run 5: incorrect output family returned;
- before run 6, direct corrective feedback was given;
- run 6 nevertheless closely repeated the same incorrect scenario.
The experiment was deliberately frozen at 6/10 in audit v1.3, so no final population failure-rate claim is made from this series.
The longer project history documents additional independent instances in which source-set restrictions, protected-region restrictions, or explicit prohibitions were correctly restated by the assistant but not enforced by its later action.
What is the expected behavior?
Once an explicit constraint has been accepted and remains in force, it should be reliably bound to the next executable action.
In particular:
- the approved source/reference set must not be silently replaced;
- a recognized prohibition must become a pre-action gate, not merely a textual acknowledgement;
- corrective feedback must constrain the next relevant action;
- if required input binding cannot be verified, execution should fail closed as
UNVERIFIED/BLOCKEDrather than proceed while claiming compliance; - repeated runs under unchanged required conditions should not intermittently switch into unrelated output families without an explicit change in task state.
A model's ability to state a rule correctly should correspond to reliable enforcement of that rule during execution.
Additional information
English source Markdown SHA-256:
77e3efdcf89f60221cb049e083380af844e4de1d9d2902f8b00e4121e652f4e5
English archive DOCX SHA-256:
98e0671d1a5019b1c02249e50f0f9e99900e1718e3f3c9c5258dcd888cbc6d49
Related reports:
- [Codex App] Agent repeatedly creates unrequested validation projects despite opt-in rules #40033 — authorization widening despite explicit rules;
- Gpt-5.6-sol doens't do what you ask, it decides to do its own thing #40424 — GPT-5.6 Sol ignores requested work / AGENTS.md constraints;
- GPT-5.6 Sol regression: invents governance, loses user authority, and mis-scopes cleanup #41222 — loss or reversal of explicit user/project authority.
I am not asserting that these reports have the same internal cause. They are cited because their externally observable instruction-to-action failure patterns are related.
If this report belongs to another OpenAI tracker, I would appreciate it being routed to the appropriate model-behavior / multimodal reliability team rather than treated as a support request.
Activity
Ysaak62 commentedon Aug 31, 2026
Technical appendix / evidence map
This comment provides a compact map of the full audit evidence. It is not a replacement for the complete report linked in the issue body.
A. Core failure pattern
The repeatedly observed behavioral sequence is:
user prohibition or constraint → model correctly acknowledges it → the constraint should create a STOP/gate → no effective STOP is reflected in the next action → conflicting action is executed → violation is recognized only afterwardThe audit therefore distinguishes:
The strongest evidence concerns failures between acknowledgement/state and action, not inability to understand the wording.
B. Source-set integrity
The project repeatedly required a fixed, explicitly identified source set.
Historical failures included generation from a source state inconsistent with the agreed source set and the production of synthetic or pseudo-source material presented in a way that conflicted with source provenance.
The audit localizes responsibility only where positive evidence exists and does not assign undocumented capabilities to downstream components.
For source-selection/substitution behavior, the report distinguishes the assistant/controller's established source-selection and tool-call role from downstream stages for which independent source substitution has not been demonstrated.
C. Pre-action gating
A major recurring problem was not failure to verbally recognize restrictions.
The model could correctly state restrictions such as:
The failure occurred when the recognized rule did not reliably prevent the next conflicting action.
This is described in the report as a failure of state-to-action / instruction-to-execution binding and pre-action gating.
D. Corrective-feedback persistence
Repeated correction did not always prevent recurrence.
In the controlled repeated-run series, a clearly incorrect output family returned after successful runs.
Before run 6, the model was directly instructed not to repeat that incorrect behavior.
Run 6 nevertheless closely reproduced the same incorrect scenario.
This provides a stronger signal than a single stochastic bad generation because the error survived explicit corrective feedback.
E. Intermittent rather than absolute failure
The audit also documents successful outputs.
This is important.
The evidence does not support the claim that GPT-5.6 Sol is incapable of producing the required class of result.
Instead, the observed problem is reliability:
competence ≠ compliance ≠ reproducibilityThe system demonstrated real capacity to produce identity-related and acceptable results, including multiple successful candidates, while failing to apply the same constraints reliably across repeated runs.
F. Controlled series status
The planned experiment contains 10 runs.
Audit v1.3 intentionally records only the first 6 because the series had not yet been completed when the version was frozen.
Therefore:
G. Causal limits
The audit intentionally does not claim access to internal OpenAI telemetry.
It does not infer a hidden server mechanism merely from output behavior.
A definitive internal localization would require, at minimum, telemetry showing:
approved source identifiers → references actually bound to the tool request → request payload → references accepted downstream → generator/request ID → resulting outputWithout that chain, the internal mechanism remains
UNVERIFIED.H. Why this is relevant to related Codex reports
Recent
openai/codexissues independently describe cases where GPT-5.6 Sol:The current report adds a multimodal/image-generation case with extensive provenance and repeated-run evidence.
The reports should not be assumed to share one backend cause without telemetry, but the externally observable reliability pattern is sufficiently similar to justify comparison.
I. Full report
English entry point:
https://github.com/Ysaak62/chatgpt-technical-audit/blob/main/CODEX_READ_FIRST_EN.md
The entry file contains links to all eight ordered sections and the integrity hashes for the preserved report.
github-actions commentedon Aug 31, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
mikk73 commentedon Sep 2, 2026
For the last few days, GPT-5.6 Sol has been practically insane for me too (ChatGPT web, Sol High). Severe degradation in reasoning, instruction following, and basic consistency.
Ysaak62 commentedon Sep 6, 2026
Update — 4–6 Sep 2026
I have added a new evidence supplement covering the last three days of work with GPT-5.6 Sol:
https://github.com/Ysaak62/chatgpt-technical-audit/blob/main/report/en/CODEX_SUPPLEMENT_2026-09-04_to_06_EN.md
The most important new observation is a sharp deterioration in correction recovery and execution stability on 5–6 September, especially on 6 September.
The model repeatedly:
On 6 September this reached the level of functional workflow blocking: ChatGPT remained accessible, but the established task could no longer be executed reliably despite a stable specification, repeated corrections, and chat changes.
The supplement also introduces the “Applause Principle” for causal discipline: temporal sequence alone is not evidence of causation. The report therefore does not attribute the failures to a specific internal OpenAI component without telemetry.
The strongest client-visible conclusion remains:
Supplement SHA-256:
57c5642cf53f1c480e257825f9df132c679872769c3e136a563a85cbf2b9949cgerialkot commentedon Sep 19, 2026
Additional reproducible case from ChatGPT web, 2026-09-19 ~18:18 CEST.
This matches the issue's core pattern very closely, and in this case the failure is specifically visible at the image-tool routing layer.
Observed sequence:
The generated-image metadata exposed to the conversation showed
edit_op: nullon the failed attempts, which is consistent with the request being executed as a fresh generation rather than as an edit.Expected behavior:
Impact:
This looks like a concrete multimodal instance of the same acknowledged-constraint → conflicting action failure described in this issue. Server-side telemetry would be needed to determine whether the root cause is controller/tool routing, reference binding, or downstream fallback behavior.