Python Skills Every AI Engineer Needs
Learn the Python engineering skills that turn AI prototypes into reliable services through a typed extraction example with validation, tests, evaluation, and failure handling.
Calling a model is the uncertain middle of an AI feature, not the whole system. Production-minded Python supplies the contracts, failure handling, deterministic checks, tests, evaluation, and operational boundaries around that call. This guide builds those capabilities around one compact invoice-extraction service: text enters, a replaceable provider returns structured data, and the application either produces a validated result or takes an explicit review path.
What production Python adds around an AI call
Deterministic code contains uncertain output instead of pretending uncertainty has disappeared.
An AI response may vary even when the surrounding application must behave predictably. Keep request parsing, authorization, identifier handling, schema validation, business rules, and error translation deterministic. Put the provider call behind a narrow boundary. This makes it possible to test the software without a network request and evaluate model behavior with a different tool.
Many preventable failures originate at data, validation, and service boundaries rather than in the model call itself. A JSON parser only proves that braces and values are syntactically valid. A domain model must still establish that required fields exist, types are correct, and values meet the application's rules.
These implementation boundaries complement the broader sequencing in the remote backend engineering roadmap; this article stays focused on the Python mechanics around an AI capability.
- Deterministic components: typed input, explicit errors, schema validation, business checks, serialization, and routing.
- Uncertain component: the external model or provider that proposes extracted values.
- Test boundary: replace the provider with a fake for exact software assertions.
- Evaluation boundary: replay labeled cases against recorded or live provider outputs and classify task failures.
A compact typed extraction service
Request and result contracts, a provider Protocol, and explicit application errors fit in one inspectable module.
Pydantic models validate data at entry and exit. Strict mode prevents convenient coercions from quietly changing the provider's meaning, while extra='forbid' catches unexpected fields. The provider is a typing.Protocol, so an adapter only needs to implement extract(text) and does not inherit framework code.
The service accepts an already validated DocumentRequest, invokes the provider once, validates its JSON as InvoiceFields, then constructs an ExtractionResult with the trusted document identifier. The provider cannot overwrite that identifier. Camel-case aliases keep the external JSON contract readable while Python attributes remain snake_case.
The example deliberately avoids a hard-coded model SDK. A real adapter may call a hosted API or local model, while tests supply a fake with no key, network, or paid service. A thin FastAPI route could validate DocumentRequest and call extract_invoice, but HTTP is not required to understand or run the core design.
app/extraction.pyfrom __future__ import annotations
from typing import Protocol
from pydantic import BaseModel, ConfigDict, Field, ValidationError
class DocumentRequest(BaseModel):
model_config = ConfigDict(strict=True, extra="forbid", str_strip_whitespace=True)
document_id: str = Field(min_length=1)
text: str = Field(min_length=1)
class InvoiceFields(BaseModel):
model_config = ConfigDict(strict=True, extra="forbid", populate_by_name=True)
vendor: str = Field(min_length=1)
total: float = Field(gt=0)
requires_review: bool = Field(alias="requiresReview")
class ExtractionResult(BaseModel):
model_config = ConfigDict(strict=True, extra="forbid", populate_by_name=True)
document_id: str = Field(alias="documentId")
vendor: str
total: float
requires_review: bool = Field(alias="requiresReview")
class ExtractionProvider(Protocol):
def extract(self, text: str) -> str:
"""Return a JSON object containing proposed invoice fields."""
class ProviderTimeout(RuntimeError):
pass
class ProviderFailure(RuntimeError):
pass
class ReviewRequired(ValueError):
def __init__(self, problems: list[str]) -> None:
self.problems = problems
super().__init__("; ".join(problems))
def extract_invoice(
request: DocumentRequest, provider: ExtractionProvider
) -> ExtractionResult:
try:
raw_output = provider.extract(request.text)
except TimeoutError as exc:
raise ProviderTimeout("Extraction provider timed out") from exc
except Exception as exc:
raise ProviderFailure("Extraction provider failed") from exc
try:
fields = InvoiceFields.model_validate_json(raw_output, strict=True)
except ValidationError as exc:
problems = [
f"{'.'.join(map(str, error['loc']))}: {error['msg']}"
for error in exc.errors()
]
raise ReviewRequired(problems) from exc
return ExtractionResult(
document_id=request.document_id,
vendor=fields.vendor,
total=fields.total,
requires_review=fields.requires_review,
)Expected output
{
"documentId": "invoice-001",
"vendor": "Example Supplies",
"total": 125.5,
"requiresReview": false
}Handle model and API failures explicitly
Every failure category needs a deliberate path; catching Exception and continuing destroys evidence.
A timeout is translated into ProviderTimeout, allowing an HTTP boundary or worker to apply its own retry or response policy. Other adapter failures become ProviderFailure. The browser never needs the provider's raw exception or credentials, and the application can log a safe failure category with a request identifier.
Malformed JSON and schema-invalid JSON both become ReviewRequired, but the validation problems remain attached. In the required failure case, the provider returns valid JSON without total. Pydantic rejects the missing field; the service neither inserts zero nor guesses an amount. This is the practical difference between valid JSON and valid domain data.
A requiresReview value supplied by the model is itself strictly typed. Deterministic business checks can add further reasons—for example, an unsupported currency or non-positive total—without changing the provider adapter. Keep retries outside this core function and limit them to failures known to be transient.
| Failure | Detection | Result |
|---|---|---|
| Timeout | Provider raises TimeoutError | Translate to ProviderTimeout; caller decides bounded retry or failure response |
| Malformed structured output | model_validate_json cannot parse it | ReviewRequired retains validation details |
| Missing total | InvoiceFields requires total | Reject; never synthesize an amount |
| Provider dependency failure | Adapter raises another exception | Translate to ProviderFailure |
| Domain validation failure | Strict type or field constraint fails | Explicit review/error path |
Test deterministic behavior
A fake provider turns the service into ordinary, exact Python tests.
pytest's plain assertions are enough here. The happy path verifies the complete serialized result. Invalid request input fails before a provider can run. A provider response that omits total proves the review path and preserves the field-level problem. A timeout test verifies translation without waiting for a real clock or network.
These are software tests, not claims about model quality. They prove that known inputs and failures move through deterministic code correctly. Add integration tests for a real adapter separately, with credentials and network behavior kept outside the unit-test suite.
tests/test_extraction.pyimport json
import pytest
from pydantic import ValidationError
from app.extraction import (
DocumentRequest, ProviderTimeout, ReviewRequired, extract_invoice,
)
class FakeProvider:
def __init__(self, output: dict | None = None, error: Exception | None = None):
self.output = output
self.error = error
def extract(self, text: str) -> str:
if self.error:
raise self.error
return json.dumps(self.output)
def test_happy_path():
provider = FakeProvider({
"vendor": "Example Supplies",
"total": 125.5,
"requiresReview": False,
})
result = extract_invoice(
DocumentRequest(document_id="invoice-001", text="Invoice text"),
provider,
)
assert result.model_dump(by_alias=True) == {
"documentId": "invoice-001",
"vendor": "Example Supplies",
"total": 125.5,
"requiresReview": False,
}
def test_invalid_request_is_rejected():
with pytest.raises(ValidationError):
DocumentRequest(document_id="invoice-001", text="")
def test_missing_total_requires_review():
provider = FakeProvider({"vendor": "Example Supplies", "requiresReview": True})
with pytest.raises(ReviewRequired) as error:
extract_invoice(DocumentRequest(document_id="invoice-002", text="Invoice"), provider)
assert any(problem.startswith("total:") for problem in error.value.problems)
def test_timeout_is_translated():
provider = FakeProvider(error=TimeoutError())
with pytest.raises(ProviderTimeout, match="timed out"):
extract_invoice(DocumentRequest(document_id="invoice-003", text="Invoice"), provider)Evaluate model behavior separately
Unit tests protect software contracts; labeled cases inspect whether proposed fields are correct.
An evaluation case stores document text, a recorded provider output, and expected fields. The runner first exercises the same strict service boundary, then compares vendor, total, and review behavior. This small fixture is inspectable rather than statistically representative: it demonstrates mechanics and failure categories, not an accuracy percentage.
case-02 contains syntactically valid JSON but omits total, so it reports FAIL missing_total. case-03 passes schema validation but proposes the wrong vendor, so it reports FAIL vendor_mismatch. This separation shows why schema validity is necessary but cannot establish extraction correctness.
evals/cases.json[
{
"id": "case-01",
"documentId": "invoice-001",
"text": "Example Supplies invoice total 125.50",
"providerOutput": {"vendor": "Example Supplies", "total": 125.5, "requiresReview": false},
"expected": {"vendor": "Example Supplies", "total": 125.5, "requiresReview": false}
},
{
"id": "case-02",
"documentId": "invoice-002",
"text": "Northwind invoice with unreadable total",
"providerOutput": {"vendor": "Northwind", "requiresReview": true},
"expected": {"vendor": "Northwind", "total": 80.0, "requiresReview": true}
},
{
"id": "case-03",
"documentId": "invoice-003",
"text": "Contoso invoice total 49.00",
"providerOutput": {"vendor": "Wrong Vendor", "total": 49.0, "requiresReview": false},
"expected": {"vendor": "Contoso", "total": 49.0, "requiresReview": false}
},
{
"id": "case-04",
"documentId": "invoice-004",
"text": "Tailspin invoice total 72.25; vendor text is uncertain",
"providerOutput": {"vendor": "Tailspin", "total": 72.25, "requiresReview": true},
"expected": {"vendor": "Tailspin", "total": 72.25, "requiresReview": true}
}
]evals/run_eval.pyimport json
from pathlib import Path
from app.extraction import DocumentRequest, ReviewRequired, extract_invoice
class RecordedProvider:
def __init__(self, output: dict):
self.output = output
def extract(self, text: str) -> str:
return json.dumps(self.output)
def classify(case: dict) -> str:
try:
result = extract_invoice(
DocumentRequest(document_id=case["documentId"], text=case["text"]),
RecordedProvider(case["providerOutput"]),
).model_dump(by_alias=True)
except ReviewRequired as error:
return "missing_total" if any("total" in item for item in error.problems) else "invalid_output"
expected = case["expected"]
if result["vendor"] != expected["vendor"]:
return "vendor_mismatch"
if result["total"] != expected["total"]:
return "total_mismatch"
if result["requiresReview"] != expected["requiresReview"]:
return "review_mismatch"
return "PASS"
for case in json.loads(Path(__file__).with_name("cases.json").read_text()):
result = classify(case)
print(f"{case['id']} {result if result == 'PASS' else f'FAIL {result}'}")Expected output
case-01 PASS
case-02 FAIL missing_total
case-03 FAIL vendor_mismatch
case-04 PASSUse a production-minded project structure
Organize by responsibility so frameworks and providers can change without rewriting domain behavior.
The compact article keeps contracts and service code together for readability. As the system grows, split domain models, provider contracts/adapters, the application service, and the optional API boundary. Tests mirror deterministic modules; evaluation data and runners remain separate because their assertions answer a different question.
For portfolio scope and presentation, continue with Python project ideas for an AI resume rather than turning this service example into a project catalog.
project treeapp/
domain.py # request/result models and domain errors
provider.py # Protocol plus model/API adapters
service.py # extraction orchestration and business checks
api.py # optional thin HTTP boundary
tests/
test_service.py # deterministic service behavior
evals/
cases.json # labeled inputs and recorded outputs
run_eval.py # field checks and failure categories- Domain models describe accepted data without importing an HTTP or model SDK.
- Provider adapters own external API details, timeouts, and response acquisition.
- The application service coordinates the provider, validation, and business rules.
- An API module translates HTTP requests and application errors; it should not contain extraction logic.
- Tests use fakes for exact behavior, while evaluation compares task outputs against labels.
Add operational safeguards
Observability should explain a failure without exposing document contents or secrets.
Assign or accept a request identifier at the transport boundary and include it in structured logs, traces, and safe error responses. Record the operation, provider name, timeout category, validation-failure category, and duration where useful. OpenTelemetry's Python APIs support spans and attributes, but instrumentation does not decide which data is safe to record.
Keep provider credentials in a secret manager or deployment environment, validate configuration at startup, and set explicit connect/read or total-call timeouts in the adapter. If retries are appropriate, bound them and make duplicate work safe. Do not log raw documents, full model prompts/responses, credentials, authorization headers, personal data, or validation values that may reproduce sensitive source text.
Notebooks remain useful for exploration and inspecting evaluation cases. When an application or another engineer depends on the logic, move stable functions into importable modules, keep the notebook as a caller, and protect the modules with tests.
Tips
- Use consistent failure-category names so logs and evaluation reports can be grouped.
- Keep raw provider errors server-side; expose a stable application error to callers.
- Separate model-quality signals from service-health signals such as timeouts and validation failures.
- Document the command that runs tests and evaluation, plus required local dependencies.
Capability checklist
You can turn an AI prototype into a reliable service when you can demonstrate each boundary.
- Model strict request and result contracts with field constraints and controlled aliases.
- Define a provider interface and replace it with a fake in tests.
- Translate timeouts and dependency failures into explicit application errors.
- Reject malformed or incomplete structured output while retaining validation problems.
- Keep deterministic business checks outside the model prompt and provider adapter.
- Write exact tests for application behavior and separate labeled evaluations for model behavior.
- Classify evaluation failures by field or review-path reason instead of hiding them in one score.
- Use structured, privacy-conscious logs and explicit configuration/timeouts.
- Promote stable notebook logic into modules when other systems depend on it.
What to build next
Extend one boundary at a time and keep the executable feedback loop short.
Start by copying the three canonical artifacts into a small Python 3.12 project, install Pydantic and pytest, and run the four tests plus the evaluation runner. Then add one provider adapter behind ExtractionProvider. Keep the recorded evaluation outputs so changes to prompts, models, or adapters can be reviewed against the same labeled cases.
If your next question is how this capability fits into end-to-end workflows, use the AI automation engineer guide. If the question is how to present the evidence to employers, use the AI engineer resume guide or review the author's current work on the About page.
Examples
- Add an unsupported-currency business rule and a test that confirms it becomes an explicit review reason.
- Add a provider adapter with an explicit timeout, then write an integration test that is opt-in rather than part of the network-free unit suite.
- Add one labeled edge case after every observed evaluation failure; do not convert a tiny fixture into an unsupported accuracy claim.
FAQ
Which Python skills matter most for AI engineering?
Strong boundaries matter more than memorizing libraries: typed data models, validation, explicit errors, testable provider interfaces, exact software tests, separate model evaluation, and safe operational instrumentation.
Why use Pydantic instead of parsing JSON into a dictionary?
JSON parsing checks syntax. A strict Pydantic model also checks required fields, types, constraints, and unexpected fields, producing structured validation errors that can drive an explicit review path.
Are pytest cases enough to evaluate an AI model?
No. Unit tests prove deterministic application behavior against controlled fakes. Model evaluation compares proposed outputs with labeled task expectations and reports quality failures such as a wrong vendor or missing total.
Should the example start as a FastAPI application?
Not necessarily. Keep domain and service code independent first. Add FastAPI as a thin boundary when HTTP delivery is required so request handling does not become the application architecture.
Sources
Primary and authoritative sources reviewed for this article.
- Python typing.Protocol documentation
Official reference for structural subtyping and protocol classes used at the provider boundary.
- Pydantic strict mode
Official documentation for strict validation behavior.
- Pydantic models
Official documentation for model configuration, validation, fields, and serialization.
- FastAPI request body documentation
Official guide to typed request-body validation at an optional HTTP boundary.
- pytest assertion documentation
Official guide to assertions used in deterministic tests.
- OpenTelemetry Python instrumentation
Official guidance for creating spans and recording attributes in Python.
Conclusion
Reliable AI services are built by making uncertainty visible and surrounding it with ordinary, inspectable software. Run the example, force the missing-total case, and add one business rule before adding another framework. Then choose the next project that lets you demonstrate the same boundaries with your own problem.