Critical software often uses languages models see less

OpenAI has published research on coding in languages with less public code and documentation than Python or JavaScript. The work focuses on COBOL, Fortran, Visual Basic, OCaml and other languages that still support banking, payments, weather forecasting and scientific systems. Maintainers often face incomplete documentation and business rules encoded only in programs written by people who have left.

Better model performance could help teams investigate, test and modernise those systems. It also raises the cost of misplaced confidence. Legacy code frequently contains rounding behaviour, file formats and failure handling that look accidental but express important operational requirements. A rewrite can compile and still change money, eligibility or safety outcomes.

MultiLangBench tests building a database engine

OpenAI created MultiLangBench, which asks a model to build a SQLite-compatible database engine across 20 languages. The task requires organising a codebase, managing state and errors, adding features and preserving existing functionality. Database files provide a shared interface so implementations in different languages can face comparable tests.

GPT-6 Astra achieved an average test pass rate of 87.4 per cent, compared with 26.8 per cent for GPT-5.6 Sol. OpenAI reports gains in every language tested. GPT-6 Luna reached 40.3 per cent at an estimated 78 per cent lower API cost than GPT-5.6 Sol, showing that model selection can trade capability against expense.

Individual languages still reveal uneven difficulty

The published lower-cost comparison includes gains from 10.9 to 17.1 per cent for COBOL, 25.6 to 34.1 for OCaml, 21.4 to 31.4 for Fortran and 27.5 to 41.6 for Visual Basic. Those absolute results are a reminder that an overall average can hide large gaps in the languages most important to a particular organisation.

OpenAI is releasing the evaluation setup and test suite on GitHub. That allows developers to inspect what is measured and run models against their own priorities. Teams should add compiler versions, platform-specific behaviour, large files, dependencies and representative maintenance tasks before treating the benchmark as evidence for production work.

COBOL migration is judged by observable parity

A second evaluation asks models to migrate COBOL programs to C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte parity in observable output. That bar is stricter than code that merely builds, and it catches differences such as fixed-point arithmetic discarding excess decimals unless rounding is explicitly requested.

GPT-6.1 Sol reached 86.6 per cent accuracy at low reasoning effort, compared with 5.4 per cent for GPT-5.6 Sol, while OpenAI estimates the newer run cost 39 per cent less. At higher reasoning effort, accuracy reached 97.9 per cent. Near-saturation on this evaluation does not mean every enterprise codebase is solved; real systems include databases, batch schedules, hardware and undocumented external contracts.

Modernisation needs dual running and domain review

A safe migration begins with characterisation tests around the existing system, not with generated replacement code. Teams need production-like records, boundary values and known historical incidents. The original and replacement should run side by side long enough to detect rare behavioural differences, with domain experts deciding whether a discrepancy is a bug or an essential rule.

AI can accelerate inventory, test generation, explanation and mechanical translation, but it should not become the sole source of truth about code it is replacing. Generated tests can repeat the model’s misunderstanding. Independent fixtures, reconciliation and staged cutover remain necessary, particularly in finance, government and scientific workloads.

The research broadens coding-model evaluation

Most coding benchmarks favour popular languages and self-contained tasks. OpenAI’s evaluations make language coverage and behaviour preservation more visible, while publishing cost alongside accuracy. That is useful for organisations deciding whether a stronger model is justified for a difficult repository or whether a cheaper model can handle routine work.

The practical opportunity is not automatic replacement of legacy systems. It is giving scarce maintainers better tools to understand unfamiliar code, create tests and explore migration options. Teams should reproduce the results on their own programs, record every model and prompt version and keep experts accountable for final behaviour. Progress is significant, but critical modernisation still succeeds through evidence, not generated confidence.

Operational test

Before production use, teams should document the exact product or model version, supported region, source date and evaluation conditions. They should repeat representative tests after any model, compiler, connector, policy or infrastructure change. Published capability and performance claims are useful starting points, but confidence must come from evidence collected in the organisation’s own environment. A named owner, measurable acceptance threshold, incident path and tested fallback turn a promising announcement into a service that can be governed over time.

Evaluation should cover ordinary work, boundary cases and deliberate misuse rather than a polished demonstration alone. Results need to be sliced by language, user role, data sensitivity and consequence so a strong average cannot hide a serious failure. Teams should also estimate the human work required to review exceptions and maintain the surrounding controls. If that effort is omitted, an apparently efficient feature may simply move cost and risk into operations, security or frontline staff who were not included in the original business case.