[FEATURE] Desktop: "screenshot / mockup → working code" workflow with built-in live preview (like CUE / code-to-UI) #41361

Activity

1. github-actions

   assigned

   Brendonovich

   on Aug 9, 2026
2. liudongyan13701205717-source commented on Aug 15, 2026

   

   liudongyan13701205717-source

   on Aug 15, 2026

   Author

   More actions

   Still highly relevant — a built-in live preview for screenshot/mockup→code would be a major Desktop differentiator (like CUE / code-to-UI flows). Happy to test on Windows Desktop once a prototype exists. +1.
3. liudongyan13701205717-source commented on Oct 1, 2026

   

   liudongyan13701205717-source

   on Oct 1, 2026

   Author

   More actions

   This was generated by AI during triage.

   Reviving this with a sharper claim than my +1 above, because the framing decides what gets built.

   The first frame is not the hard part. Turning a screenshot into plausible HTML is commodity now — any vision model does it. If that is the whole feature, it duplicates the model and adds nothing.

   The scarce part is the loop, and today the loop is user-driven. The model can see a mockup only when a human attaches it; it cannot see what it just produced. So today's actual flow is: generate → user opens preview → user screenshots → user re-attaches → model guesses at the diff. Every iteration pays a human round-trip, and the model is being asked to compare against a description of a screenshot rather than the screenshot itself.

   What makes that agent-driven is narrow and buildable:

   1. A preview pane the agent can write into.
   2. A screenshot tool scoped to that pane, so the agent can capture its own output.
   3. Viewport and DPR pinned to match the source capture — this is the part that quietly decides whether the feature works. A diff between a 1440-wide desktop capture and a 1280-wide render is dominated by layout shift from the resize, so the model spends its turn chasing phantom differences. Pin the viewport and DPR at capture time or the signal is noise.
   4. A pixel-diff score fed back as text, not just a picture — "3.1% of pixels differ, concentrated in the header band" is actionable; a second screenshot is not, because the model has to re-derive what changed.

   With those four, render → screenshot → diff → fix becomes a loop the agent runs unattended, and the convergence quality is bounded by the harness rather than by how well a human narrates a screenshot. That is a real differentiator, and it reuses a pixel-diff harness that visual-QA tooling already needs.

   One caveat worth stating up front: a high pixel score is necessary but not sufficient — it will happily converge on a layout that matches the screenshot and fails every interaction. So the loop needs interaction assertions alongside the visual diff, or it optimizes toward a beautiful still image.

   Happy to test on Windows Desktop, including the viewport/DPR matching, which is where I would expect a first implementation to get quietly wrong.
4. neriousy commented on Oct 21, 2026

   

   neriousy

   on Oct 21, 2026

   Member

   More actions

   Thanks for the detailed write-up. Most of this loop already works in Desktop: attach the screenshot, ask the agent to build the page, and it can open the result in Desktop's built-in browser and take its own screenshots to compare against the mockup and iterate. If you want a one-step /screenshot-code flow, you can package the instructions as a custom command (https://opencode.ai/v2/docs/commands) or a skill (https://opencode.ai/v2/docs/skills) and share it. We're not planning to build a dedicated screenshot-to-code workflow into opencode, so we're closing this as not planned.
5. neriousy

   closed this as not plannedon Oct 21, 2026