AI / Core OpenAI Codex Application Fundamentals Interview Questions
What is the OpenAI Evals framework and why is evaluation critical for production applications?
Evals (evaluations) are automated tests that measure an LLM application's quality, accuracy, and reliability. OpenAI provides both an Evals API (for running evaluations programmatically) and the OpenAI Evals Framework (open-source, run locally). Without systematic evals, you cannot confidently iterate on prompts, upgrade models, or know if your application is regressing.
from openai import OpenAI client = OpenAI() # Using the Evals API: # 1. Create a dataset for evaluation: dataset = client.evals.datasets.create( name="code-review-eval", items=[ {"input": "Review: def add(a,b): return a-b", "expected": "Bug: subtraction used instead of addition"}, {"input": "Review: import os; os.system(user_input)", "expected": "Security: OS injection vulnerability"}, ] ) # 2. Create and run an eval: eval_run = client.evals.runs.create( name="code-reviewer-v2-test", model="gpt-5.5", data_source={"type": "dataset", "id": dataset.id}, grader_config={ "type": "llm_as_judge", "model": "gpt-5.4-mini", # fast/cheap grader "criteria": "Does the review correctly identify all bugs and security issues?" } ) print(f"Score: {eval_run.result.score}") # Open-source evals (run locally): # pip install evals # oaieval gpt-5.5 my-code-review-eval # Compare two models: # oaieval gpt-5.5 my-eval # oaieval gpt-5.4 my-eval
When to evaluate:
- Before deploying any prompt change to production
- Before upgrading to a new model version
- After any significant change to the application logic
- Periodically in production to detect model drift
Good evals catch regressions before users do and give you confidence to upgrade models without fear of breaking existing behaviour.
What is the primary purpose of running evals before upgrading an OpenAI model in production?
What role does an 'LLM as judge' grader play in the OpenAI Evals API?
More Related questions...
