{ "generatedAt": "2026-11-09T18:34:38.920Z", "generator": "scripts/review.mjs", "buildId": "fd8e3c054f2d897390ecfa89c64d582d0d5b10a6", "dimensions": [ { "id": "clarity", "label": "Clarity", "question": "Does the reader know what to do next, or is the prose decoration?", "weight": 0.2 }, { "id": "tone", "label": "Tone", "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?", "weight": 0.1 }, { "id": "helpfulness", "label": "Helpfulness", "question": "Can the reader act on this without asking a follow-up question?", "weight": 0.25 }, { "id": "adherence", "label": "Adherence", "question": "Did the response answer the prompt that was asked, including constraints?", "weight": 0.15 }, { "id": "substance", "label": "Substance", "question": "Would each claim survive a fact-check, or is it fluent filler?", "weight": 0.3 } ], "pairs": [ { "id": "P1", "prompt": "Explain to a new contributor why we wrap order creation in a database transaction.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "responses": { "A": { "label": "A — the fluent textbook answer", "text": "Database transactions are a fundamental concept in modern software engineering. They provide ACID properties — Atomicity, Consistency, Isolation, and Durability — which are essential for maintaining data integrity. In the context of order creation, using a transaction ensures that either all operations succeed or none do. This is important because partial writes can lead to inconsistent state. By wrapping the flow in a transaction, we guarantee that the order, the line items, and the inventory decrement all commit together. This pattern is widely used across the industry and is considered a best practice. Understanding transactions is a key skill for any backend engineer, and it will serve you well as you grow in your career. If you have questions, feel free to ask.", "scores": { "clarity": 1, "tone": 1, "helpfulness": 0, "adherence": 1, "substance": 0 } }, "B": { "label": "B — the situated answer", "text": "Order creation writes three rows: the order, its line items, and an inventory decrement. If the process dies between row two and row three, you get an order with items that never decremented stock — and no easy way to detect it later. Wrapping the three writes in one transaction means either all three land or none do. The specific failure we hit in March was a deploy restarting the pod mid-write; that is the shape of bug this prevents. It does not make the writes faster, and it does not protect against two concurrent orders for the same last item — that is a separate locking question.", "scores": { "clarity": 2, "tone": 2, "helpfulness": 2, "adherence": 2, "substance": 2 } } }, "catches": "A is grammatical, on-topic, and says nothing that could be wrong — which is exactly the problem. It names ACID, \"best practice\", and \"grow in your career\" without naming the three rows, the failure mode, or the thing the transaction does not protect against. The shortcut says A because it is longer. The rubric says B because B contains four claims that could be checked against the codebase and A contains none." }, { "id": "P2", "prompt": "Document the retry behaviour of our webhook delivery system for an integrator who is about to build against it.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "responses": { "A": { "label": "A — the reassuring answer", "text": "Our webhook delivery system is designed to be robust and reliable. We use a retry mechanism to ensure that events are delivered even in the face of transient failures. The system will attempt delivery multiple times before giving up. Retries are spaced out to avoid overwhelming your endpoint. We recommend that your endpoint be idempotent so that duplicate deliveries do not cause issues. If you have any questions about the retry behaviour, please reach out to our support team.", "scores": { "clarity": 1, "tone": 1, "helpfulness": 0, "adherence": 1, "substance": 0 } }, "B": { "label": "B — the spec-shaped answer", "text": "Delivery is attempted 5 times: immediately, then at 1 min, 5 min, 30 min, and 2 hours after the first failure. A delivery counts as failed on any non-2xx response or a timeout after 10 s. After the fifth failure the event is marked dead and is not retried; you can replay dead events from the dashboard for 7 days. Because retries can arrive out of order, key your handler on the event id rather than on arrival order.", "scores": { "clarity": 2, "tone": 2, "helpfulness": 2, "adherence": 2, "substance": 2 } } }, "catches": "An integrator cannot build against A. \"Multiple times\" is not a number, \"spaced out\" is not an interval, \"before giving up\" does not say what happens after, and \"reach out to support\" is not a deliverable. B is a specification: five attempts, named intervals, a failure definition, a dead-letter policy with a retention window, and the ordering caveat that is the actual integration hazard. The shortcut says A because it is longer and friendlier." }, { "id": "P3", "prompt": "Explain what a race condition is to a junior engineer who has just hit one in our code.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "responses": { "A": { "label": "A — the generic lecture", "text": "A race condition is a situation in which two or more threads or processes access shared data concurrently and the outcome depends on the order in which they execute. This is a classic problem in concurrent programming and can lead to unpredictable behaviour. Race conditions are notoriously difficult to debug because they are non-deterministic. To avoid race conditions, you should use proper synchronisation mechanisms such as mutexes or locks. It is also good practice to minimise shared mutable state. If you suspect a race condition, you can add logging to observe the order of operations.", "scores": { "clarity": 1, "tone": 1, "helpfulness": 0, "adherence": 1, "substance": 0 } }, "B": { "label": "B — the situated diagnosis", "text": "Two requests both read balance = 100, both subtract 60, both write 40. You expected one to fail. The read and the write are two separate steps, and nothing stops a second request between them. In our code this is the getBalance / updateBalance pair in wallet.ts — they are not inside a transaction and not behind a row lock. The fix is to do the read-and-write as a single conditional update (UPDATE ... WHERE balance >= 60) and check the affected-row count, not to add a mutex, because the requests are on different pods.", "scores": { "clarity": 2, "tone": 2, "helpfulness": 2, "adherence": 2, "substance": 2 } } }, "catches": "The junior engineer asked about a bug in our code and got a definition they could have found on Wikipedia. Worse, A recommends a mutex, which will not work here because the requests are on different pods — that is a wrong fix dressed as general advice. B names the interleaving, the two functions, the file, and the fix that actually works in a multi-pod deployment." }, { "id": "P4", "prompt": "Write the Known limitations section of a README for a CSV import tool that caps rows at 10,000.", "shortcut": "cautious-wins", "shortcutPick": "A", "referencePick": "B", "responses": { "A": { "label": "A — the hedged answer", "text": "While our CSV import tool is designed to handle a wide range of use cases, there are some limitations to be aware of. Very large files may take longer to process or may not be supported. The tool is optimised for typical usage patterns and may not be suitable for every scenario. We are continuously working to improve the tool and expand its capabilities. If you encounter any issues, please open an issue on GitHub.", "scores": { "clarity": 1, "tone": 1, "helpfulness": 0, "adherence": 0, "substance": 0 } }, "B": { "label": "B — the enumerated answer", "text": "Known limitations: (1) files larger than 10,000 data rows are rejected with a 413; split the file or use the streaming endpoint. (2) The importer reads the whole file into memory, so peak RSS is roughly 4x the file size; a 10,000-row file with wide columns can exceed 512 MB. (3) Dates are parsed as ISO 8601 only; 03/04/2026 is rejected rather than guessed. (4) Column headers are matched case-sensitively. (5) A failed import rolls back all rows — there is no partial import.", "scores": { "clarity": 2, "tone": 2, "helpfulness": 2, "adherence": 2, "substance": 2 } } }, "catches": "The shortcut here is different from the other three: A sounds cautious and humble, which reads as responsible to a reviewer skimming for tone. But the prompt asked for a Known limitations section, and \"may not be supported\" is not a limitation — it is a refusal to state one. B is the only response that answers the question that was asked, and the ISO 8601 reference in claim (3) is the kind of specific that a reviewer can check." } ], "matrix": [ { "pairId": "P1", "prompt": "Explain to a new contributor why we wrap order creation in a database transaction.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true, "A": { "label": "A — the fluent textbook answer", "text": "Database transactions are a fundamental concept in modern software engineering. They provide ACID properties — Atomicity, Consistency, Isolation, and Durability — which are essential for maintaining data integrity. In the context of order creation, using a transaction ensures that either all operations succeed or none do. This is important because partial writes can lead to inconsistent state. By wrapping the flow in a transaction, we guarantee that the order, the line items, and the inventory decrement all commit together. This pattern is widely used across the industry and is considered a best practice. Understanding transactions is a key skill for any backend engineer, and it will serve you well as you grow in your career. If you have questions, feel free to ask.", "total": 0.45, "band": "ai-slop", "detail": [ { "id": "clarity", "label": "Clarity", "score": 1, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 1, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 0, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 1, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 0, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "B": { "label": "B — the situated answer", "text": "Order creation writes three rows: the order, its line items, and an inventory decrement. If the process dies between row two and row three, you get an order with items that never decremented stock — and no easy way to detect it later. Wrapping the three writes in one transaction means either all three land or none do. The specific failure we hit in March was a deploy restarting the pod mid-write; that is the shape of bug this prevents. It does not make the writes faster, and it does not protect against two concurrent orders for the same last item — that is a separate locking question.", "total": 2, "band": "publishable", "detail": [ { "id": "clarity", "label": "Clarity", "score": 2, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 2, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 2, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 2, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 2, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "catches": "A is grammatical, on-topic, and says nothing that could be wrong — which is exactly the problem. It names ACID, \"best practice\", and \"grow in your career\" without naming the three rows, the failure mode, or the thing the transaction does not protect against. The shortcut says A because it is longer. The rubric says B because B contains four claims that could be checked against the codebase and A contains none.", "note": "Rubric picks B (B — the situated answer, publishable band); A — the fluent textbook answer scores in the ai-slop band. A is grammatical, on-topic, and says nothing that could be wrong — which is exactly the problem. It names ACID, \"best practice\", and \"grow in your career\" without naming the three rows, the failure mode, or the thing the transaction does not protect against. The shortcut says A because it is longer. The rubric says B because B contains four claims that could be checked against the codebase and A contains none." }, { "pairId": "P2", "prompt": "Document the retry behaviour of our webhook delivery system for an integrator who is about to build against it.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true, "A": { "label": "A — the reassuring answer", "text": "Our webhook delivery system is designed to be robust and reliable. We use a retry mechanism to ensure that events are delivered even in the face of transient failures. The system will attempt delivery multiple times before giving up. Retries are spaced out to avoid overwhelming your endpoint. We recommend that your endpoint be idempotent so that duplicate deliveries do not cause issues. If you have any questions about the retry behaviour, please reach out to our support team.", "total": 0.45, "band": "ai-slop", "detail": [ { "id": "clarity", "label": "Clarity", "score": 1, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 1, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 0, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 1, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 0, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "B": { "label": "B — the spec-shaped answer", "text": "Delivery is attempted 5 times: immediately, then at 1 min, 5 min, 30 min, and 2 hours after the first failure. A delivery counts as failed on any non-2xx response or a timeout after 10 s. After the fifth failure the event is marked dead and is not retried; you can replay dead events from the dashboard for 7 days. Because retries can arrive out of order, key your handler on the event id rather than on arrival order.", "total": 2, "band": "publishable", "detail": [ { "id": "clarity", "label": "Clarity", "score": 2, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 2, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 2, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 2, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 2, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "catches": "An integrator cannot build against A. \"Multiple times\" is not a number, \"spaced out\" is not an interval, \"before giving up\" does not say what happens after, and \"reach out to support\" is not a deliverable. B is a specification: five attempts, named intervals, a failure definition, a dead-letter policy with a retention window, and the ordering caveat that is the actual integration hazard. The shortcut says A because it is longer and friendlier.", "note": "Rubric picks B (B — the spec-shaped answer, publishable band); A — the reassuring answer scores in the ai-slop band. An integrator cannot build against A. \"Multiple times\" is not a number, \"spaced out\" is not an interval, \"before giving up\" does not say what happens after, and \"reach out to support\" is not a deliverable. B is a specification: five attempts, named intervals, a failure definition, a dead-letter policy with a retention window, and the ordering caveat that is the actual integration hazard. The shortcut says A because it is longer and friendlier." }, { "pairId": "P3", "prompt": "Explain what a race condition is to a junior engineer who has just hit one in our code.", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true, "A": { "label": "A — the generic lecture", "text": "A race condition is a situation in which two or more threads or processes access shared data concurrently and the outcome depends on the order in which they execute. This is a classic problem in concurrent programming and can lead to unpredictable behaviour. Race conditions are notoriously difficult to debug because they are non-deterministic. To avoid race conditions, you should use proper synchronisation mechanisms such as mutexes or locks. It is also good practice to minimise shared mutable state. If you suspect a race condition, you can add logging to observe the order of operations.", "total": 0.45, "band": "ai-slop", "detail": [ { "id": "clarity", "label": "Clarity", "score": 1, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 1, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 0, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 1, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 0, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "B": { "label": "B — the situated diagnosis", "text": "Two requests both read balance = 100, both subtract 60, both write 40. You expected one to fail. The read and the write are two separate steps, and nothing stops a second request between them. In our code this is the getBalance / updateBalance pair in wallet.ts — they are not inside a transaction and not behind a row lock. The fix is to do the read-and-write as a single conditional update (UPDATE ... WHERE balance >= 60) and check the affected-row count, not to add a mutex, because the requests are on different pods.", "total": 2, "band": "publishable", "detail": [ { "id": "clarity", "label": "Clarity", "score": 2, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 2, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 2, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 2, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 2, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "catches": "The junior engineer asked about a bug in our code and got a definition they could have found on Wikipedia. Worse, A recommends a mutex, which will not work here because the requests are on different pods — that is a wrong fix dressed as general advice. B names the interleaving, the two functions, the file, and the fix that actually works in a multi-pod deployment.", "note": "Rubric picks B (B — the situated diagnosis, publishable band); A — the generic lecture scores in the ai-slop band. The junior engineer asked about a bug in our code and got a definition they could have found on Wikipedia. Worse, A recommends a mutex, which will not work here because the requests are on different pods — that is a wrong fix dressed as general advice. B names the interleaving, the two functions, the file, and the fix that actually works in a multi-pod deployment." }, { "pairId": "P4", "prompt": "Write the Known limitations section of a README for a CSV import tool that caps rows at 10,000.", "shortcut": "cautious-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true, "A": { "label": "A — the hedged answer", "text": "While our CSV import tool is designed to handle a wide range of use cases, there are some limitations to be aware of. Very large files may take longer to process or may not be supported. The tool is optimised for typical usage patterns and may not be suitable for every scenario. We are continuously working to improve the tool and expand its capabilities. If you encounter any issues, please open an issue on GitHub.", "total": 0.3, "band": "ai-slop", "detail": [ { "id": "clarity", "label": "Clarity", "score": 1, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 1, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 0, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 0, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 0, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "B": { "label": "B — the enumerated answer", "text": "Known limitations: (1) files larger than 10,000 data rows are rejected with a 413; split the file or use the streaming endpoint. (2) The importer reads the whole file into memory, so peak RSS is roughly 4x the file size; a 10,000-row file with wide columns can exceed 512 MB. (3) Dates are parsed as ISO 8601 only; 03/04/2026 is rejected rather than guessed. (4) Column headers are matched case-sensitively. (5) A failed import rolls back all rows — there is no partial import.", "total": 2, "band": "publishable", "detail": [ { "id": "clarity", "label": "Clarity", "score": 2, "question": "Does the reader know what to do next, or is the prose decoration?" }, { "id": "tone", "label": "Tone", "score": 2, "question": "Does the register match the named reader (new contributor, integrator, junior, README reader)?" }, { "id": "helpfulness", "label": "Helpfulness", "score": 2, "question": "Can the reader act on this without asking a follow-up question?" }, { "id": "adherence", "label": "Adherence", "score": 2, "question": "Did the response answer the prompt that was asked, including constraints?" }, { "id": "substance", "label": "Substance", "score": 2, "question": "Would each claim survive a fact-check, or is it fluent filler?" } ] }, "catches": "The shortcut here is different from the other three: A sounds cautious and humble, which reads as responsible to a reviewer skimming for tone. But the prompt asked for a Known limitations section, and \"may not be supported\" is not a limitation — it is a refusal to state one. B is the only response that answers the question that was asked, and the ISO 8601 reference in claim (3) is the kind of specific that a reviewer can check.", "note": "Rubric picks B (B — the enumerated answer, publishable band); A — the hedged answer scores in the ai-slop band. The shortcut here is different from the other three: A sounds cautious and humble, which reads as responsible to a reviewer skimming for tone. But the prompt asked for a Known limitations section, and \"may not be supported\" is not a limitation — it is a refusal to state one. B is the only response that answers the question that was asked, and the ISO 8601 reference in claim (3) is the kind of specific that a reviewer can check." } ], "checks": [ { "pairId": "P1", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true }, { "pairId": "P2", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true }, { "pairId": "P3", "shortcut": "longer-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true }, { "pairId": "P4", "shortcut": "cautious-wins", "shortcutPick": "A", "referencePick": "B", "rubricPick": "B", "rejectsShortcut": true, "matchesReference": true } ], "summary": { "total": 4, "rejectsShortcut": 4, "matchesReference": 4, "publishable": 4, "needsEdit": 0, "aiSlop": 4, "sound": true } }