Export a Runnable Eval Suite Before the Read-Only Date

The read-only date is the real export deadline

If your AI release gate runs on OpenAI Evals, the date that matters is not the shutdown date. Existing OpenAI Evals become read-only on October 31, 2026. On June 3, 2026, OpenAI marked reusable prompt objects, Agent Builder, and the Evals platform for retirement. OpenAI's Evals platform, Agent Builder, and reusable prompt objects are scheduled to shut down on November 30, 2026.

That timeline looks generous only if the suite can already produce a runnable artifact. A release gate that cannot export a runnable configuration after that date is a design failure, even if the dashboard still exists. A dashboard may still show history, but history is not a release decision. For a product designer, the issue is not just infrastructure. Users do not see the eval dashboard. They see the feature that ships, the answer that fails, and the rollback that arrives too late. If the release gate depends on a platform that will stop accepting new runs, the team needs a portable version of the decision. That version has to be runnable, not merely archived. An archived score is evidence. A runnable suite is a control. The redesign is simple: make the gate portable before the platform stops accepting work.

A completed run is the hidden prerequisite

Teams treat the export as a file transfer. It is closer to a handoff. The structure can move, but the ability to rerun it after the platform stops accepting work does not.

An OpenAI Evals suite must have a completed run before it can be exported as a runnable configuration. If the suite has only drafts, partial runs, or a stale score, the export path is not ready. The visible deadline hides the prerequisite. A product team can see the calendar date. It can also miss the state the suite must reach before the export becomes useful. The missing state is not visible in a calendar invite or a migration note. That gap is where release gates quietly stop being gates.

The export leaves the judgment behind

An export is a partial migration. OpenAI Evals exports do not include graders in the exported configuration. That omission is easy to miss until the new pipeline scores a prompt and the number does not mean what the old dashboard meant. A grader is the judgment layer. Without it, the dataset is portable, but the release decision is not. A score without a stable judge is a number, not a gate. Promptfoo is the replacement OpenAI recommends for the Evals platform.

Moving the suite there is straightforward. Rebuilding the judgment is where teams lose continuity, because proving that the new scoring logic still says the same thing about the same behavior is the hard part.

Model-based judges make the problem sharper. Manually recreated graders, especially LLM-as-a-judge graders, should be validated before they are used for regression decisions in an OpenAI Evals migration. A rebuilt judge that scores the same behavior differently is an unstable release gate. The team needs to compare the rebuilt judge against the baseline before trusting it to block or approve a launch.

A five-step checklist keeps the gate honest

The failure mode is narrow, so the checklist is short. Complete it before the read-only date. Each item is a check you can finish in a minute, not a project. It exists to keep the release decision honest after the platform stops accepting new runs. Order matters: the export depends on the run, and the new score depends on the graders.

• Complete one baseline run and save its score.
• Export the completed run as a runnable Promptfoo configuration.
• Rebuild every grader outside the export.
• Revalidate LLM-as-a-judge graders before using them for regression decisions.
• Verify the exported suite reproduces the baseline score in the new pipeline.

Reproduction is the final validation. If the new pipeline cannot reproduce the baseline score, do not use it to block a launch.