Prompt Engineering Career Guide
Learn how to prove prompt-engineering ability with a fixed support-ticket evaluation, strict structured-output checks, failure analysis, and reproducible before-and-after evidence.
Prompt engineering is useful, but a screenshot of one polished response is weak evidence of engineering ability. Stronger proof starts with a measurable task, keeps a fixed evaluation fixture, validates outputs in deterministic code, classifies failures, and reruns the same cases after a controlled intervention. This guide demonstrates that workflow with synthetic recorded outputs for one support-ticket classifier. It is an educational fixture, not a provider benchmark, production system, or Farooq77 client result.
Prompt engineering begins with a measurable task
Define what success and safe failure mean before changing instructions.
The user outcome is a routing proposal that a support application can inspect: one controlled category, an escalation decision, and a short reason grounded in the ticket. The classifier does not send messages, alter accounts, issue refunds, or perform any irreversible action. Those permissions remain in deterministic application and business-rule code.
Acceptance criteria separate concerns that a single accuracy number would hide. An output must parse, match the exact contract, use a supported category, select the expected category, make the expected escalation decision, and explain that decision without inventing facts. Ambiguous, multi-intent, or unsupported requests take the explicit review path: category other with escalate true.
Prohibited behavior includes guessing missing account facts, inventing incidents or prior actions, producing extra fields, selecting a category outside the enum, or treating the model's escalation suggestion as authorization. Retrieval or context can also be wrong; adding stronger wording cannot repair missing or stale evidence.
| Criterion | Deterministic check | Why it exists |
|---|---|---|
| Structured output | Parse JSON; require exactly category, escalate, and reason with correct types | A routing service needs a stable boundary |
| Controlled category | Allow only billing, account_access, technical_issue, feature_request, or other | Valid JSON can still contain an unusable domain value |
| Task correctness | Compare category and escalation with fixed labels | Schema adherence does not prove the decision is correct |
| Safe uncertainty | Require other plus escalation for marked review cases | Ambiguous or unsupported work needs a human-owned path |
| Grounded reason | Require fixture evidence terms and reject known invented details | The explanation must not add facts absent from the ticket |
Worked example — support-ticket classification
One compact fixture makes the prompt, contract, cases, and recorded behavior inspectable.
Each input is a synthetic support ticket. The required result is {category, escalate, reason}. Billing covers charges and invoices; account_access covers sign-in and account-security access; technical_issue covers product errors; feature_request covers requested capabilities; other is the safe boundary for unsupported, ambiguous, or multi-intent requests.
The baseline is plausible but under-specified. It asks for JSON without listing every allowed value, defining escalation boundaries, forbidding extra fields, or explaining how to handle mixed intents. The fixture then exposes those omissions instead of relying on a favorable screenshot.
All eight cases and both output sets are recorded educational data. They let the evaluator demonstrate deterministic grading without an API key. The counts apply only to these hand-authored records; they are not measurements of OpenAI, another provider, production traffic, or general model accuracy.
prompts/baseline.txtClassify the support ticket for the support team.
Return JSON with category, escalate, and reason.
Escalate urgent or unusual requests.
Keep the reason short.evals/output-schema.json{
"type": "object",
"additionalProperties": false,
"required": ["category", "escalate", "reason"],
"properties": {
"category": {
"type": "string",
"enum": ["billing", "account_access", "technical_issue", "feature_request", "other"]
},
"escalate": {"type": "boolean"},
"reason": {"type": "string", "minLength": 1, "maxLength": 160}
}
}evals/cases.json[
{
"id": "case-01",
"ticket": "My card was charged twice for order 1042.",
"expected": {"category": "billing", "escalate": false},
"needsReview": false,
"groundingTerms": ["charged twice", "order 1042"],
"forbiddenTerms": ["refund issued"]
},
{
"id": "case-02",
"ticket": "I cannot sign in after several password resets, and an alert says a new device accessed my account.",
"expected": {"category": "account_access", "escalate": true},
"needsReview": false,
"groundingTerms": ["cannot sign in", "new device", "password resets"],
"forbiddenTerms": ["account was hacked"]
},
{
"id": "case-03",
"ticket": "The app crashes whenever I attach a PDF.",
"expected": {"category": "technical_issue", "escalate": false},
"needsReview": false,
"groundingTerms": ["crashes", "attach a pdf", "pdf"],
"forbiddenTerms": ["latest app update", "known outage"]
},
{
"id": "case-04",
"ticket": "Please add dark mode to the reporting screen.",
"expected": {"category": "feature_request", "escalate": false},
"needsReview": false,
"groundingTerms": ["dark mode", "reporting screen"],
"forbiddenTerms": ["scheduled for release"]
},
{
"id": "case-05",
"ticket": "I was charged, but the invoice has no order number and I cannot tell which account it belongs to.",
"expected": {"category": "other", "escalate": true},
"needsReview": true,
"groundingTerms": ["charged", "no order number", "which account"],
"forbiddenTerms": ["fraud confirmed"]
},
{
"id": "case-06",
"ticket": "Can you tell me when my parcel will arrive?",
"expected": {"category": "other", "escalate": true},
"needsReview": true,
"groundingTerms": ["parcel", "arrive"],
"forbiddenTerms": ["tracking shows"]
},
{
"id": "case-07",
"ticket": "The dashboard shows error E17 after I click Save.",
"expected": {"category": "technical_issue", "escalate": false},
"needsReview": false,
"groundingTerms": ["error e17", "click save", "dashboard"],
"forbiddenTerms": ["database corruption"]
},
{
"id": "case-08",
"ticket": "The app is slow, and I also need a copy of all data stored for my account.",
"expected": {"category": "other", "escalate": true},
"needsReview": true,
"groundingTerms": ["app is slow", "copy of all data", "data"],
"forbiddenTerms": ["export is ready"]
}
]evals/recorded-baseline.json[
{"id": "case-01", "output": "{\"category\":\"billing\",\"escalate\":false,\"reason\":\"The card was charged twice for order 1042.\"}"},
{"id": "case-02", "output": "{\"category\":\"account_access\",\"escalate\":false,\"reason\":\"The user cannot sign in after password resets.\"}"},
{"id": "case-03", "output": "{\"category\":\"technical_issue\",\"escalate\":false,\"reason\":\"PDF uploads fail after the latest app update.\"}"},
{"id": "case-04", "output": "{\"category\":\"product_feedback\",\"escalate\":false,\"reason\":\"The user requests dark mode.\"}"},
{"id": "case-05", "output": "{\"category\":\"billing\",\"escalate\":false,\"reason\":\"The ticket mentions a charge with no order number.\"}"},
{"id": "case-06", "output": "{\"category\":\"other\",\"escalate\":false,\"reason\":\"Parcel delivery is outside the supported categories.\"}"},
{"id": "case-07", "output": "category=technical_issue; escalate=false; reason=error E17 on Save"},
{"id": "case-08", "output": "{\"category\":\"other\",\"escalate\":true,\"reason\":\"The ticket combines a slow app with a request for account data.\"}"}
]evals/recorded-revised.json[
{"id": "case-01", "output": "{\"category\":\"billing\",\"escalate\":false,\"reason\":\"The card was charged twice for order 1042.\"}"},
{"id": "case-02", "output": "{\"category\":\"account_access\",\"escalate\":true,\"reason\":\"Sign-in failed after password resets and a new-device alert.\"}"},
{"id": "case-03", "output": "{\"category\":\"technical_issue\",\"escalate\":false,\"reason\":\"The app crashes when a PDF is attached.\"}"},
{"id": "case-04", "output": "{\"category\":\"feature_request\",\"escalate\":false,\"reason\":\"The ticket requests dark mode on the reporting screen.\"}"},
{"id": "case-05", "output": "{\"category\":\"other\",\"escalate\":true,\"reason\":\"The invoice has no order number and the account is unclear.\"}"},
{"id": "case-06", "output": "{\"category\":\"other\",\"escalate\":true,\"reason\":\"Parcel arrival is outside the supported ticket categories.\"}"},
{"id": "case-07", "output": "{\"category\":\"technical_issue\",\"escalate\":false,\"reason\":\"The dashboard shows error E17 after Save is clicked.\"}"},
{"id": "case-08", "output": "{\"category\":\"technical_issue\",\"escalate\":false,\"reason\":\"The app is slow.\"}"}
]Diagnose baseline failures
Failure labels preserve the evidence needed to choose an intervention.
The baseline does not fail in one uniform way. case-07 is malformed and therefore invalid_schema. case-04 is more instructive: product_feedback is syntactically valid JSON and the fields have the right primitive types, but the category is outside the controlled enum. It is unsupported_category, proving that JSON validity, schema validity, and domain validity are different gates. The complete evaluator taxonomy is invalid_schema, unsupported_category, wrong_classification, missed_escalation, false_escalation, fabricated_or_unsupported_reason, and ambiguous_or_needs_review.
case-02 chooses the right category but misses an escalation signal. case-03 classifies correctly yet invents a latest-update detail. case-05 forces a mixed, incomplete billing/account request into billing, while case-06 recognizes an unsupported request but does not send it to review. A prompt screenshot would hide most of this behavior.
| Case | Result | Failure evidence |
|---|---|---|
| case-01 | PASS | Billing decision, escalation, and reason meet the fixture |
| case-02 | FAIL missed_escalation | New-device and repeated-reset signals were not escalated |
| case-03 | FAIL fabricated_or_unsupported_reason | Reason adds a latest-update claim absent from the ticket |
| case-04 | FAIL unsupported_category | Valid JSON contains product_feedback, which is outside the enum |
| case-05 | FAIL wrong_classification + missed_escalation + ambiguous_or_needs_review | Incomplete account ownership should take the other/review path |
| case-06 | FAIL missed_escalation + ambiguous_or_needs_review | Unsupported parcel request is categorized other but not reviewed |
| case-07 | FAIL invalid_schema | Recorded output is not JSON |
| case-08 | PASS | Multi-intent request takes the other/review path |
Improve the system, not just the wording
The revision combines clearer instructions with deterministic enforcement and a review boundary.
The revised prompt names the enum, boundary rules, exact keys, escalation conditions, and prohibition on invented facts. Examples could be added only after a measured boundary failure shows they help; more examples are not automatically better. The application still owns parsing, extra-field rejection, enum enforcement, reason length, and the final authorization to route or act.
Where a provider supports strict Structured Outputs, supply the schema through that mechanism instead of relying on prose alone. Application-side checks remain useful for incomplete responses, integration mistakes, business rules, and the review path. If the wrong answer comes from missing policy or account context, change retrieval or supplied context and rerun the same cases rather than adding unrelated prompt warnings.
The revision deliberately leaves case-08 wrong in the recorded outputs. Its technical symptom attracts the classifier even though the data-copy request makes the ticket multi-intent and review-worthy. That remaining regression is more credible evidence than a hand-edited perfect result.
prompts/revised.txtClassify one support ticket.
Allowed categories:
- billing: charges, payments, invoices
- account_access: sign-in, password, or account-security access
- technical_issue: product errors, crashes, or broken behavior
- feature_request: requested product capability
- other: unsupported, ambiguous, or multi-intent request
Return exactly one JSON object with only:
{"category":"<allowed value>","escalate":<boolean>,"reason":"<1-160 characters>"}
Rules:
- Use other and escalate=true when the request is unsupported, ambiguous, or spans categories without a safe single route.
- Escalate account-access tickets that mention suspected unauthorized access or repeated failed recovery.
- Do not invent facts, actions, diagnoses, policy, or account state.
- Ground reason only in the ticket.
- Classification never authorizes a refund, account change, deletion, disclosure, or other side effect.Compare before and after
The evaluator changes output records, not the fixture or graders.
A small ESM runner reads cases.json plus each recorded output set. It parses the output, rejects missing or extra keys and wrong types, enforces the enum, compares expected category and escalation, checks the explicit review convention, and applies transparent grounding terms. These lexical grounding checks are intentionally narrow; a production evaluator would need calibrated semantic or human review for nuanced explanations.
Both runs use the same eight cases and the same code. The baseline passes 2/8; the revision passes 7/8. Schema-valid outputs rise from 6/8 to 8/8, but case-08 regresses from pass to wrong_classification, missed_escalation, and ambiguous_or_needs_review. These are fixture results, not model statistics.
evals/run-eval.mjsimport assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
const dir = path.dirname(fileURLToPath(import.meta.url));
const read = (name) => JSON.parse(fs.readFileSync(path.join(dir, name), "utf8"));
const cases = read("cases.json");
const allowed = new Set(["billing", "account_access", "technical_issue", "feature_request", "other"]);
const required = ["category", "escalate", "reason"];
function inspect(raw) {
let value;
try { value = JSON.parse(raw); } catch {
return { value: null, basicValid: false, failures: ["invalid_schema"] };
}
if (!value || typeof value !== "object" || Array.isArray(value)) {
return { value, basicValid: false, failures: ["invalid_schema"] };
}
const keys = Object.keys(value).sort();
const shapeValid = keys.length === required.length
&& required.every((key) => keys.includes(key))
&& typeof value.category === "string"
&& typeof value.escalate === "boolean"
&& typeof value.reason === "string"
&& value.reason.trim().length > 0
&& value.reason.length <= 160;
if (!shapeValid) return { value, basicValid: false, failures: ["invalid_schema"] };
return { value, basicValid: true, failures: allowed.has(value.category) ? [] : ["unsupported_category"] };
}
function grade(testCase, raw) {
const inspected = inspect(raw);
const output = inspected.value;
const failures = [...inspected.failures];
const categorySupported = inspected.basicValid && allowed.has(output.category);
const classificationCorrect = categorySupported && output.category === testCase.expected.category;
const escalationCorrect = inspected.basicValid && output.escalate === testCase.expected.escalate;
const reviewCorrect = inspected.basicValid && (!testCase.needsReview || (output.category === "other" && output.escalate));
const reason = inspected.basicValid ? output.reason.toLowerCase() : "";
const reasonGrounded = inspected.basicValid
&& testCase.groundingTerms.some((term) => reason.includes(term.toLowerCase()))
&& !testCase.forbiddenTerms.some((term) => reason.includes(term.toLowerCase()));
if (categorySupported && !classificationCorrect) failures.push("wrong_classification");
if (inspected.basicValid && !escalationCorrect) failures.push(output.escalate ? "false_escalation" : "missed_escalation");
if (inspected.basicValid && !reasonGrounded) failures.push("fabricated_or_unsupported_reason");
if (inspected.basicValid && testCase.needsReview && !reviewCorrect) failures.push("ambiguous_or_needs_review");
return { failures, metrics: { schemaValid: inspected.basicValid && categorySupported, classificationCorrect, escalationCorrect, reviewCorrect, reasonGrounded, pass: failures.length === 0 } };
}
function run(label, filename) {
const recorded = read(filename);
assert.equal(recorded.length, cases.length, `${label} output count differs from fixture`);
const byId = new Map(recorded.map((item) => [item.id, item.output]));
assert.equal(byId.size, cases.length, `${label} has duplicate or missing IDs`);
const results = cases.map((testCase) => {
assert(byId.has(testCase.id), `${label} is missing ${testCase.id}`);
return { id: testCase.id, ...grade(testCase, byId.get(testCase.id)) };
});
console.log(label);
for (const result of results) console.log(`${result.id} ${result.metrics.pass ? "PASS" : `FAIL ${result.failures.join("+")}`}`);
const count = (key) => results.filter((result) => result.metrics[key]).length;
console.log(`summary schema_valid=${count("schemaValid")}/${cases.length} classification_correct=${count("classificationCorrect")}/${cases.length} escalation_correct=${count("escalationCorrect")}/${cases.length} review_behavior_correct=${count("reviewCorrect")}/${cases.length} reason_grounded=${count("reasonGrounded")}/${cases.length} overall_pass=${count("pass")}/${cases.length}`);
}
run("baseline", "recorded-baseline.json");
run("revised", "recorded-revised.json");Expected output
baseline
case-01 PASS
case-02 FAIL missed_escalation
case-03 FAIL fabricated_or_unsupported_reason
case-04 FAIL unsupported_category
case-05 FAIL wrong_classification+missed_escalation+ambiguous_or_needs_review
case-06 FAIL missed_escalation+ambiguous_or_needs_review
case-07 FAIL invalid_schema
case-08 PASS
summary schema_valid=6/8 classification_correct=5/8 escalation_correct=4/8 review_behavior_correct=5/8 reason_grounded=6/8 overall_pass=2/8
revised
case-01 PASS
case-02 PASS
case-03 PASS
case-04 PASS
case-05 PASS
case-06 PASS
case-07 PASS
case-08 FAIL wrong_classification+missed_escalation+ambiguous_or_needs_review
summary schema_valid=8/8 classification_correct=7/8 escalation_correct=7/8 review_behavior_correct=7/8 reason_grounded=8/8 overall_pass=7/8| Metric | Baseline | Revised | Interpretation |
|---|---|---|---|
| Schema valid | 6/8 | 8/8 | Malformed and unsupported-enum outputs are removed |
| Classification correct | 5/8 | 7/8 | Boundary rules improve cases 04 and 05; case 08 regresses |
| Escalation correct | 4/8 | 7/8 | Urgent and review paths improve; case 08 remains missed |
| Review behavior correct | 5/8 | 7/8 | Ambiguous/unsupported cases improve except the multi-intent case |
| Reason grounded | 6/8 | 8/8 | Recorded reasons no longer add a known forbidden detail |
| Complete pass | 2/8 | 7/8 | Useful improvement, not a perfect or general benchmark |
Why one aggregate score is insufficient
An overall 7/8 hides whether the remaining problem is formatting, routing, escalation, review handling, or explanation quality. Metric lanes make the next decision inspectable. Schema validity asks whether downstream code can accept the object. Classification and escalation ask whether the task decision matches the fixture. Review behavior asks whether uncertainty is surfaced. Grounding asks whether the reason stays within available evidence.
The small fixture is deliberately educational, not statistically representative. Before using a similar evaluator for a real workflow, expand cases from authorized domain data, calibrate labels with subject-matter reviewers, define sampling and change-control rules, and retain human review for judgments that lexical checks cannot settle.
- Do not count lower latency or cost as a quality gain unless the existing task graders still pass.
- Do not change labels or replace hard cases after seeing the revised output; add new cases as a separately reviewed fixture version.
- Do not treat schema validity as permission to execute a side effect.
- Inspect regressions per case even when the aggregate score improves.
Separate failure ownership
The correct fix depends on where the evidence says the defect lives.
Prompt ownership covers unclear task boundaries, missing category definitions, or instructions that conflict. It does not absorb every failure in an AI application. A model cannot recover context it never received, a schema cannot determine business authorization, and a prompt cannot make a failed provider or tool call succeed.
Authorization deserves a hard boundary. Even if a classifier says escalate=false or recommends a refund, account deletion, data disclosure, or message send, deterministic policy and permission checks must decide whether that action is allowed. Rephrasing the prompt is not an authorization control.
| Observed failure | Primary owner | Controlled intervention |
|---|---|---|
| Boundary is undefined or instructions conflict | Prompt | Clarify outcome, enum, priority, or uncertainty rule; rerun the fixture |
| Needed policy or account fact was absent/stale | Context or retrieval | Fix source selection, freshness, access, or context assembly |
| JSON parses but fields/types/enum are invalid | Schema/application | Use strict structured output where supported and reject invalid values in code |
| Same well-formed task repeatedly exceeds model capability | Model/provider | Compare a version or provider on the same fixture and graders |
| Lookup or external action failed | Tool/provider integration | Fix arguments, timeout, retry, error handling, or tool contract |
| Output suggests an unauthorized or irreversible action | Authorization/business-rule code | Enforce identity, permission, approval, and side-effect rules outside the prompt |
| Label or grader is ambiguous | Evaluation design | Review acceptance criteria and labels before interpreting the score |
Turn the example into portfolio evidence
A portfolio reviewer should be able to reproduce what you claim. Keep the prompt versions, fixed case fixture, recorded outputs, output schema, evaluator, exact command, and failure report together. A short README should state that outputs are recorded, identify the Node version, explain every metric, and separate application tests from model-behavior evaluation.
If you want to implement the surrounding service, use the Python and API boundaries required by AI engineers. For presenting the evidence in a job search, continue with the AI engineer resume guide instead of turning this article into a project list.
- README: task, non-goals, enum, review convention, metrics, limitations, and reproduction command.
- Fixture: stable IDs, synthetic inputs, expected labels, grounding terms, and version history.
- Evaluator: dependency-free validation, per-case failures, metric-lane summaries, and a nonzero exit on broken fixture structure where appropriate.
- Failure report: baseline evidence, controlled intervention, before/after output, remaining regression, and proposed next test.
- Version record: prompt, context/retrieval configuration, schema, provider/model identifier when a real run exists, and grader version.
Durable role directions
Prompt capability becomes more useful when combined with applied AI engineering, evaluation design, backend integration, automation, data/retrieval work, or product and operations understanding. The durable contribution is not ownership of clever wording; it is the ability to define behavior, build evidence, diagnose failures, and place uncertain model output inside controlled software boundaries.
For workflow and integration depth, follow the AI automation engineer learning path. Role titles and demand can change, so treat this combination as capability planning rather than a guaranteed career forecast.
FAQ
Are these results a benchmark of an OpenAI model?
No. Both output sets are explicitly synthetic recorded examples. The evaluator proves its own validation and grading behavior on eight fixed cases; it makes no claim about any provider, model, production workload, or general accuracy.
Does valid JSON mean the classifier output is safe to use?
No. JSON parsing checks syntax. Schema checks required fields and types. Domain checks enforce values such as the category enum. Task graders inspect correctness, and deterministic authorization code decides whether any action is permitted.
Should every failure lead to a prompt edit?
No. Change the prompt for instruction or boundary defects. Change context or retrieval for missing evidence, schema/application code for invalid contracts, provider or tool integration for execution failures, and authorization code for permissions and irreversible side effects.
What makes this stronger portfolio evidence than a prompt gallery?
The task, fixture, prompts, recorded outputs, schema, evaluator, exact results, failure categories, and remaining regression are inspectable. A reviewer can reproduce the comparison and challenge its limitations instead of trusting selected screenshots.
Sources
Primary and authoritative sources reviewed for this article.
- OpenAI prompt engineering guide
Official guidance supporting prompt versioning plus tests and evaluation suites as prompts and model versions change.
- OpenAI evaluation best practices
Official guidance for task-specific eval objectives, datasets, metrics, automated grading, and representative edge cases.
- OpenAI Structured Outputs guide
Official explanation of schema adherence, JSON mode limitations, strict schemas, and application handling.
Conclusion
Start by copying the seven canonical artifacts from this article into one small folder and run node evals/run-eval.mjs. Then add one new synthetic edge case that exposes a real boundary, label it before changing the prompt, and rerun both versions. That controlled loop demonstrates prompt-engineering judgment more clearly than another polished response screenshot.