Code Samples and Code Comments

The most verifiable thing you'll write, and the most likely to rot — here's how to keep samples current.

The Writers’ Table

---

Photo by Artturi Jalli on Unsplash

Code samples are technical documentation with the clearest success criterion: they run, or they don’t. A procedure can be followed loosely. A conceptual explanation can be interpreted. A code sample either executes without errors or it doesn’t, and a sample that doesn’t run destroys the reader’s trust faster than almost any other documentation failure.

This makes code samples both the most verifiable part of API and developer documentation and the most maintenance-intensive.

The Golden Rule: Test Your Samples

Every code sample in your documentation should run. Not “should theoretically run.” Should run, as verified by running it.

While this seems obvious, it is routinely ignored. The reasons are varied: the sample was written before the API was finalized, the API changed after the sample was written and nobody updated it, or the sample was written by someone who understood the concept but didn’t verify the syntax.

The fix is a testing process. For sample code in documentation, this means you run the sample in the actual environment it’s intended for, with real credentials against the real API (or a stable staging environment), and verify that it produces the expected output.

One rule for what gets published: test with real credentials and publish with obvious placeholders. A sample should show YOUR_API_KEY or <api-key>, never a working key. Real keys leak into documentation more often than anyone admits, and a leaked key on a public docs site is a security incident.

For documentation repositories that follow a docs-as-code workflow, samples can be tested in CI. A GitHub Action that runs code samples against a staging environment and fails the PR if any sample breaks is not a complex pipeline, and it prevents the class of errors where a code change breaks examples without anyone noticing.

Formatting and Syntax Highlighting

Keep your Markdown code clean and readable with two simple rules:

1. Specify language tags on code blocks. Adding identifiers like python turns on syntax highlighting. Without it, your code looks like a flat block of plain text.

2. Backtick your inline code. Whenever you mention variables (user_id), file paths (/etc/hosts), or commands (git commit) inside a sentence, wrap them in backticks so they pop out to readers scanning the page.

Writing Useful Comments

Code comments in documentation samples have one job that documentation prose doesn’t: they explain what a specific line does when prose context isn’t adjacent to the line.

The rule for code comments: comment why, not what. “Retrieves the user object” adds nothing to client.users.get(user_id), because the function name says the same thing. “The user_id must be the internal UUID, not the display name” tells the reader something the function name can’t.

That rule comes from production code, and sample comments serve a different reader. A production comment is written for the next maintainer, who knows the codebase and needs the reasoning behind a decision. A sample comment is written for someone meeting your API for the first time, who can’t tell which parts of the snippet are your product and which are ordinary language mechanics. Write for the second reader. Explain the parameter that behaves unexpectedly.

The test for what belongs in a comment rather than in prose is the paste boundary. Readers copy the code block; they don’t copy the paragraph above it. Anything that has to survive the paste goes in the comment. That covers substitution instructions like # Replace with your key, runtime constraints, and anything a reader would otherwise discover by failing.

In documentation samples, comments should flag the places where users are most likely to make mistakes or where the behavior differs from what they’d expect:

``` python
# Rate limit: 100 requests per minute per API key
# Exceeding this returns 429; implement exponential backoff
for user_id in user_ids:
    response = client.users.get(user_id=user_id)
```

This comment is valuable. It tells the reader something about behavior that isn’t visible from the code alone, and that will cause problems if they don’t know it. Note that the sample shows the pitfall without implementing the fix. That’s a deliberate trade-off: keeping the sample short enough to illustrate one thing, while the comment points to what production code needs to add.

One complication arises once you stop writing samples in the doc and start embedding them from source files. The comments travel with the code. They live in a repo, get read by engineers maintaining that file, and get published to readers who have never seen it. Settle ownership before the first embedded sample ships. If the sample files are yours to maintain, write the comments for the reader and let engineering work around it. If they belong to engineering, you need agreement that reader-facing comments aren’t clutter, because the alternative is a published sample whose comments assume internal knowledge.

Maintaining Samples Through API Changes

Code samples become outdated when APIs evolve.

The documentation maintenance problem is that code samples are embedded in prose documents, and there’s no automated inventory of all places where the deprecated get_user() function appears in documentation. This is where a docs-as-code workflow with search capabilities helps: grep -r "get_user" docs/ finds every instance.

The structural protection is keeping samples short and focused. A 10-line sample demonstrating a single operation is easier to maintain than a 50-line sample demonstrating an entire workflow. It has fewer lines to update when things change, and it’s clearer which concept each line illustrates.

When an API changes, audit your samples as part of the change documentation process, not as a separate cleanup task.

Tools for Samples

Always run the sample. Always. For a handful of snippets, do that by hand. With many code samples, you want tooling, and there are two problems you can automate: sample validity and sample maintenance.

1. The first: is the sample valid at all? If your samples are in a language with a test-friendly doc format, they can join the test suite directly. Python’s built-in doctest runs samples that include their expected output and fails when the output changes; pytest --doctest-modules folds those into a normal test run. Sphinx’s doctest extension does the same for samples embedded in Sphinx docs. Go and Rust have their own versions of the idea: go test runs Example functions and compares the result against the // Output: comment, and cargo test compiles and runs every code block in Rust doc comments. Once a sample is executable by a test runner, “does it still work” stops being a question you have to remember to ask.

2. The second is the maintenance problem from the last section: samples hand-copied into prose have no inventory and drift from the real API. The strongest fix is to stop copying code into docs at all. Tools like embedme and Snipsync (built by Temporal) keep the canonical sample in a real source file that your build already compiles and tests, then inject it into the Markdown by reference. In embedme you write a marker like <!-- embedme path/to/sample.py --> and the tool pastes in the current file contents; Snipsync uses named SNIPSTART and SNIPEND markers to pull a labeled region out of a source repo. The doc shows the code; the code lives where CI can break the build if it stops working, and a renamed parameter fails a test instead of silently rotting an example.

There’s a tradeoff, though. This adds setup and a build step. It shows value in large sample libraries and fast-moving APIs, but less so in a few stable snippets. But the principle underneath is the same: always test. A sample you can’t run in CI is a sample you’re trusting on faith.

How Many Languages?

Cover the languages your users actually use, in order of frequency. Usage data from your developer dashboard, community forum analysis, or a simple survey will tell you whether Python, JavaScript, and Java cover most of your users or whether you need Go, Ruby, and .NET as well.

Publishing samples in six languages you barely maintain is worse than publishing samples in three languages you keep current. Outdated samples in a language your users rely on undermine the documentation; missing samples in a language five users have requested is a manageable gap. Prioritize quality over breadth.

Where AI Fits (and Where It Invents)

AI is genuinely useful for samples. It will draft a snippet from an endpoint description, translate a working Python sample into JavaScript and Go, add explanatory comments, and adapt an example to a reader’s parameters. For the multi-language question just above, an AI draft is the fastest way to a first pass in a language you don’t write daily.

But code samples are the worst place to trust an unverified AI draft. A USENIX Security 2025 study generated 576,000 code samples across 16 models in Python and JavaScript, then checked every package those models recommended. 19.7% of the packages didn’t exist (Spracklen et al.). That number is the aggregate, pulled up by open-source models averaging 21.7%; the commercial models most writers actually use still hallucinated at 5.2%, about one package in twenty. The same tendency produces method names, parameters, and options that read as correct and aren’t.

What turns this from a bug into an attack is that the hallucinations repeat. 58% of the invented package names appeared more than once when the same prompt was rerun ten times, so an attacker can assemble a predictable list of names, register them on npm or PyPI with malicious code, and wait for a developer to copy an AI sample and run pip install. Seth Larson, security developer-in-residence at the Python Software Foundation, named this attack slopsquatting in April 2025, and Andrew Nesbitt of Ecosyste.ms popularized the term. It isn’t theoretical. Security researcher Bar Lanyado published an empty placeholder package under huggingface-cli, a name generative tools kept inventing. It was downloaded more than 30,000 times in three months, and Alibaba’s GraphTranslator repository recommended installing it in the README.

A hallucinated function call throws an error. A hallucinated dependency is a security hole, and if it lands in your published docs, it’s your security hole.

The rule to always test doesn’t change. It matters more. An AI-drafted sample is a claim about an API that may not be true, and running it is how you find out whether it is. Confirm that every imported package and every called method actually exist, then publish under the same placeholder rule as everything else. The model can write the sample. It can’t tell you the sample is real. That task is still yours.

Further Reading

• Tom Johnson, “Code samples” — idratherbewriting.com/learnapidoc/docapis_codesamples_bestpractices.html — the code samples topic in his Code tutorials chapter, and the most directly applicable resource here: best practices for code samples in API documentation specifically, from the same course previously cited.

• Google, “Creating sample code” — developers.google.com/tech-writing/two/sample-code — a short, practical unit on writing and maintaining correct, runnable samples; a good complement to the Golden Rule section here.

• Google, “Using LLMs in tech writing” — developers.google.com/tech-writing/two/llms — the next unit in that same course, and the closest thing to an official position on where AI drafting belongs in a documentation workflow.

• Google style guides — google.github.io/styleguide — per-language code style guides; useful when your samples need to look like idiomatic code in each language you publish, not just code that runs.

• Clean Code, Robert C. Martin — the chapter on comments is the standard treatment of the “why, not what” principle.

• Python doctest — docs.python.org/3/library/doctest.html — the standard-library way to make Python samples self-testing; the fastest on-ramp to testing samples if your docs are Python-heavy.

• Go Example functions — pkg.go.dev/testing#hdr-Examples — the same idea in Go: examples that live in the test file, run under go test, and fail when output changes.

• Rust doctests — doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html — the strictest version of executable samples: cargo test compiles and runs every code block in your doc comments.

• Snipsync — github.com/temporal-community/snipsync — and Temporal’s write-up, “How we keep our code examples fresh”, a concrete look at embedding samples from tested source files rather than hand-copying them.

• Docs as Tests: A Strategy for Resilient Technical Documentation, Manny Silva — amazon.com — the book-length argument for treating documentation as something you validate automatically rather than proofread.

• “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs” — Spracklen, Wijewickrama, Sakib, Maiti, Viswanath, and Jadliwala, USENIX Security 2025 — arxiv.org/abs/2406.10279 — the primary source for every hallucination number in the AI section above, including the per-model breakdown and the repeatability data that makes the attack practical.

What You Can Do

Find a code sample in any documentation, your own, your company’s, or a public API’s.

Try to run it. (If you can’t run it, read it carefully and identify anything that looks like it might not work.) Did it run, and if not, what was the problem? Post it in the comments.

---

Next: Words in the Machine: Intro to UX Writing — the difference between UX writing and technical writing, and why every technical writer should know both.