Capstone: From One Shop to a Multi-Branch Service
Introductions, exercises and summaries stay visible.
51.0 What this chapter gives you#
- We began with a person recording a sale. We can now name the contracts, representations, queries, transactions, copies and operational responsibilities that make a larger service understandable.
- This capstone assembles a small executable teaching system rather than adding a new layer of unexplained technology. Its purpose is to make selected correctness claims observable, including refused operations and recovery after injected failures.
- The original Mira’s Corner fixture remains unchanged: O-1042 is 17,100 paise, O-1043 is 11,550, and four agreed lines represent six units and 28,650 paise. Those are agreed line amounts, not proof of payment settlement.
- The capstone introduces separate synthetic identifiers: tenants T-A/T-B, branch B-1, products NOTE/PEN, requests R-1/R-2 and orders CAP-001/CAP-002. These are new test fixtures, not explanations of the earlier unresolved stock discrepancy.
- The implementation is an isolated Python/SQLite learning lab. It does not include a production login, payment processor, carrier, web server, replicated database, physical-failure test or legal compliance determination.
51.1 Requirements and boundaries#
51.1.1 PLAIN — in simple words#
- Before choosing machinery, define what the small service must do. It should accept a permitted order, record the agreed prices, reduce the selected stock and remember the result of a repeated request.
- Related changes must succeed together or leave no accepted order. A partially written order with fully deducted stock is not the intended outcome.
- Each tenant’s work must remain separate through the trusted helper interface. The same request or order identifier may exist independently in two tenants.
- The service also records an intention to create a local follow-up task. That intention must not disappear because an acknowledgement is lost, and repeated delivery must not create repeated local tasks.
- Not every real shop requirement belongs in the miniature model. Exclusions are stated so readers do not mistake a useful teaching implementation for a finished commercial checkout.
51.1.2 PLAIN — a picture in your head#
- Mira writes an order, marks the shelf count and leaves a task for packing. The three records describe different responsibilities but belong to one accepted operation.
- Dev repeating the same request after the answer was lost should recover the earlier result, not pretend another customer bought the same goods.
- Where the comparison breaks: the lab’s packing task is another database row. Creating that row does not physically pack a parcel, and it cannot make an uncontrolled external carrier perform an action exactly once.
51.1.3 PLAIN — a worked example#
- Initialise T-A/B-1 with ten notebooks and fifty pens. The catalogue prices are 7,550 and 2,000 paise respectively. The separate T-B fixture has its own records with the same local product codes.
- Submit R-1 for T-A order CAP-001 with two NOTE and one PEN. Its accepted total is 2×7,550 + 1×2,000 = 17,100 paise. Remaining stock is eight notebooks and forty-nine pens in that branch.
- Submit the identical R-1 again. The result remains CAP-001 at 17,100; stock remains eight and forty-nine; the outbox still contains one intention for that request.
- Submit R-1 with three notebooks instead. That is a different command under the same request identity and is rejected as a conflict. The earlier result is not overwritten.
- Submit a T-B target using a T-A-only principal. The helper rejects it before returning or changing protected T-B records. This does not test whether a real token was authenticated correctly; the principal is supplied by the trusted test harness.
51.1.4 PLAIN — what is really happening inside#
- Input validation establishes the command’s shape. Authorisation establishes its permitted scope. A database transaction then binds the accepted result, stock change, request record and outbox intention.
- A request identity connects retries to the same intended operation. The stored command representation permits comparison with later uses of that identity.
- Database constraints reject selected impossible states. Application code handles workflow decisions the schema does not express. Tests inspect both the successful path and the places where protection is intentionally absent.
51.1.5 TECHNICAL — the engineer’s version#
- The contract accepts bounded nonempty identifiers, one branch, a nonempty cart, unique product codes and strictly positive integer quantities. Python booleans are rejected as quantities even though bool is a subclass of int.
- Prices are copied from the lab’s catalogue inside the owned transaction. Request replay returns the persisted accepted result instead of repricing against a later catalogue. The example does not implement quote expiry, discounts or a separate price-approval workflow.
- The helper requires an idle connection and owns BEGIN IMMEDIATE, COMMIT and ROLLBACK. SQLite transaction and driver behaviour are explained by their documentation; the selected workflow and limits are original. [S25] [S86]
51.1.6 WORDS — remember these#
Accepted operation: a committed result under a declared contract — the business outcome represented by the transaction, not merely a received HTTP request. Scope boundary: where the guarantee applies — the trusted helper, tenant, database and supported operations included in the claim. Exclusion: a deliberately unimplemented responsibility — an explicit limit such as payment settlement or physical dispatch.
51.2 Schema and workflows#
51.2.1 PLAIN — in simple words#
- Store each fact where its meaning is clear. Catalogue price describes what is offered now; an order-line price describes what was accepted for that order.
- Stock belongs to a tenant, branch and product. An order belongs to a tenant and has lines referring to products in that same tenant.
- Request records describe replay identity. Outbox records describe intended follow-up. Inbox records describe locally accepted deliveries. These are not three copies of an order total with identical purposes.
- A separate schema for these responsibilities makes inconsistencies easier to state and test. It does not remove the need to decide which changes must share a transaction.
51.2.2 PLAIN — a picture in your head#
- Mira keeps a price board, an order ledger, a shelf register and a packing-task tray. The price board changes without rewriting yesterday’s signed order.
- A note in the tray refers to an order rather than asking the packer to reconstruct the order from today’s prices.
- Where the comparison breaks: database references can be enforced mechanically, while paper references depend on people. Neither a foreign key nor a paper number proves that the real goods moved.
51.2.3 PLAIN — a worked example#
- The lab schema uses these logical grains. A scoped key includes tenant wherever two tenants can legitimately reuse the local identity.
| Relation | One row means | Important key or boundary |
|---|---|---|
| tenants | One synthetic organisation | tenant_id |
| catalogue | Current offer for one product | tenant_id, sku |
| stock | One branch’s available quantity | tenant_id, branch_id, sku |
| orders | One accepted order | tenant_id, order_id |
| order_lines | One accepted product line | tenant_id, order_id, line_no |
| requests | One successful request result | tenant_id, request_id |
| outbox | One intended task notification | tenant_id, event_id |
| inbox | One locally processed delivery identity | tenant_id, event_id |
| tasks | One local follow-up task | tenant_id, event_id |
- CAP-001 has two order lines. Its total is derived from the accepted line quantities and prices, not recalculated from the current catalogue on every read.
- Reprice NOTE in a separate test after acceptance. Reading CAP-001 still yields 17,100 paise. A genuinely new order can use the new catalogue price under the lab’s contract.
- The schema rejects a child whose scoped parent does not exist. Test that the child cannot borrow an order identity from another tenant by omitting part of the reference.
51.2.4 PLAIN — what is really happening inside#
- The accept_order helper validates and authorises, starts its transaction, checks request identity, resolves product prices, conditionally deducts stock and inserts the related records.
- The stock UPDATE includes a sufficient-quantity predicate. A zero-row result is treated as a failure of the intended reservation, not as an accepted order with missing stock work.
- The accepted prices are protected from routine helper edits because no supported helper operation rewrites order lines. That is not the same as database-level immutability against an unrestricted connection.
- All writes use bound values. Parameterisation prevents values from being interpreted as SQL syntax; it does not establish whether the caller is allowed to make the request.
51.2.5 TECHNICAL — the engineer’s version#
- SQLite STRICT tables, explicit NOT NULL requirements, CHECK predicates and composite foreign keys form the declarative part of the model. Foreign-key enforcement is enabled and verified on helper-created connections. [S60] [S59]
- A transaction serialises this lab’s write operation under SQLite’s locking model. The result is not a proof that a corresponding PostgreSQL implementation behaves identically at every isolation level.
- The implementation accepts one owned transaction at a time per connection and fails rather than silently committing a caller’s unrelated pending work. This transaction-ownership boundary is part of its API contract.
51.2.6 WORDS — remember these#
Row grain: what one row asserts — the identity and meaning that determine valid joins and constraints. Agreed price snapshot: the price stored at acceptance — an order fact distinct from the mutable catalogue. Transaction ownership: responsibility for completion — the component entitled to commit or roll back the particular unit of work.
51.3 Correctness under retries#
51.3.1 PLAIN — in simple words#
- A retry is ambiguous unless it identifies the operation being repeated. The service needs to tell another delivery of the same command from a new command that happens to look similar.
- Store the request identity with its successful result in the same transaction as the effect. Otherwise a failure can leave either an effect with no replay record or a replay record with no effect.
- An outbox records follow-up intent atomically with the accepted order. Delivery can still be repeated, so the receiver needs its own duplicate policy.
- Exactly one local database task is a narrower claim than exactly one email, payment or physical shipment. Keep that distinction when crossing external boundaries.
51.3.2 PLAIN — a picture in your head#
- Dev sends a numbered instruction and does not hear back. He sends the same number again. Mira checks the completed-instruction register and returns the stored answer.
- Mira’s packing notification can also be delivered twice. The packer checks that notification’s number before creating a second task.
- Where the comparison breaks: the two registers may live in different databases and commit independently. A message acknowledgement cannot magically make two independent commits one indivisible event.
51.3.3 PLAIN — a worked example#
- Run accept_order for R-1 and deliberately ignore the returned result. The committed state still contains the order, stock reduction, request result and outbox intention.
- Retry R-1 with the same canonical command. The helper returns the stored result without a second deduction. Change its cart under the same key and the comparison rejects the conflict.
- Deliver the outbox event to the local consumer. The consumer commits an inbox identity and one task together. Now simulate a lost acknowledgement before the producer records delivery completion.
- Redeliver that event. The consumer sees the already-recorded identical event and does not create another task. The producer can then record a successful acknowledgement. The example observes two deliveries but one local task.
- Inject an exception after task insertion but before its consumer transaction commits. Both task and inbox insert roll back. A later delivery can therefore create the task once instead of being falsely suppressed by an inbox-only residue.
51.3.4 PLAIN — what is really happening inside#
- Idempotency requires an identity scope, a payload comparison, an effect boundary and a retention rule. A unique request key alone does not explain what happens when the payload changes or the old record expires.
- The lab’s command representation has sorted unique product lines and explicit branch/order identifiers. It compares canonical text, not only a hash, so digest equality is not treated as proof that arbitrary payloads are equal.
- Producer completion and consumer effect are distinct observations. A lost acknowledgement leaves the producer uncertain even though the consumer may already have committed.
- Retiring replay/inbox records would change the protection window. The exercise does not silently garbage-collect them and continue claiming permanent duplicate suppression.
51.3.5 TECHNICAL — the engineer’s version#
- Uniqueness and related effect writes share an owned transaction. Duplicate detection is a database-backed protocol, not a process-local set that disappears when the application restarts.
- The consumer stores the event payload with its identity and checks conflicting redelivery. An identity reused for a different task is not accepted as a harmless duplicate.
- The demonstration uses local SQLite transactions and synthetic delivery calls. It does not exercise a broker, network partition or external provider’s idempotency contract. Chapters 26 and 39 explain why those boundaries need their own design.
51.3.6 WORDS — remember these#
Idempotency scope: where a replay identity is meaningful — the tenant, operation and retention boundary governing duplicate handling. Outbox: atomically recorded delivery intent — a local record that survives with the business change but does not guarantee external completion. Inbox: the consumer’s delivery record — an identity committed with the local effect to support repeat handling. Acknowledgement loss: the result exists but the sender did not learn it — a source of uncertainty rather than proof that the action failed.
51.4 Reporting and reconciliation#
51.4.1 PLAIN — in simple words#
- A useful report states what its rows and totals mean. The same data can answer questions about orders, lines, units or agreed money, but those are not interchangeable measures.
- Keep the original teaching fixture separate from newly accepted capstone orders. Combining them without labels changes the population and makes earlier totals appear wrong.
- Reconcile at the level where mistakes can occur: identities, quantities, prices, stock changes and task counts. One matching grand total cannot detect every error.
- Report generation should preserve authorisation scope. A correct sum over the wrong tenant is still the wrong response.
51.4.2 PLAIN — a picture in your head#
- Mira compares the order ledger, shelf changes and packing tasks at closing time. Each record answers a different part of what happened.
- Two wrong prices can cancel in the total. Checking the individual lines reveals the defect the headline number hides.
- Where the comparison breaks: a database reconciliation compares stored claims, not the physical shelf itself. A physical count remains an independent observation that can disagree for several reasons.
51.4.3 PLAIN — a worked example#
- Recreate the earlier canonical fixture using its unchanged lab. Assert two orders, four lines, six units and 28,650 paise. The expected-stock-10/observed-stock-9 example remains unresolved.
- In a separate capstone database, accept CAP-001 only. Assert its two lines, three units, 17,100 paise and one outbox event. Do not add this deliberately similar example to the canonical total.
- After two deliveries of the same capstone event, assert one inbox row and one task for that scoped identity. Count delivery attempts separately from completed local effects.
- If an independent T-B order has the same local order ID, a T-A report must still include only T-A’s authorised records. Test the filter using distinguishable values, not two identical fixtures that would hide a mix-up.
51.4.4 PLAIN — what is really happening inside#
- A query produces a new relation whose grain follows its joins and grouping. Joining one order to multiple lines and multiple task records can multiply contributions unless the query controls each relationship.
- Stored totals can be checked against line sums; stock changes can be compared with accepted reservations; task identities can be compared with recorded outbox events. Each reconciliation has a time and scope boundary.
- External outcomes remain separate. An order total is not a payment settlement, and a task row is not a dispatched parcel. The schema would need additional evidence-bearing records for those claims.
51.4.5 TECHNICAL — the engineer’s version#
- The companion tests use explicit ORDER BY where result ordering matters and compare complete keys and values. SQL output order is not inferred from insertion order.
- Keep aggregates at their intended grain, using preaggregation or EXISTS where appropriate. Chapter 13’s fan-out counterexample remains a reminder that syntactically valid SQL can answer the wrong question. [S58] [S80]
- A reconciliation report records its definition and exclusions. It does not quietly turn a sum of agreed line amounts into revenue recognition, tax calculation or cash accounting.
51.4.6 WORDS — remember these#
Reconciliation population: the records intended for comparison — a defined set that prevents mixing separate fixtures or tenants. Effect count: how many accepted changes occurred — distinct from how many attempts or deliveries were observed. External outcome: an event beyond the local transaction — payment, delivery or another result needing separate evidence.
51.5 Recovery and access tests#
51.5.1 PLAIN — in simple words#
- Test interrupted work, not only successful work. A failure halfway through a multi-step change should leave a state the contract can explain.
- Test recovery into a fresh destination and compare meaningful records. The presence of a backup file is not a successful restore result.
- Test forbidden scope before and after errors. A rollback should not leave reused request context pointing at another tenant.
- Name the failure being simulated. A raised Python exception is not a power cut, and an in-memory restore is not an off-site disaster-recovery exercise.
51.5.2 PLAIN — a picture in your head#
- Mira rehearses an interrupted order by stopping after writing one page and checking that the temporary work is discarded consistently.
- She rehearses recovery by opening a copied ledger at a different desk and reading actual entries. Looking at the closed recovery box is not enough.
- Where the comparison breaks: software failures occur at several layers. Exception handling exercises the application path; physical durability depends on storage, operating-system and device promises not exercised by that same test.
51.5.3 PLAIN — a worked example#
- Inject an exception after stock deduction and before accepted-order completion. Assert that stock, orders, lines, request record and outbox return to the pre-operation state under the owned transaction.
- Retry after removing the injected failure. Assert a single accepted result and the expected remaining stock. A failed attempt did not consume the successful request key permanently in this contract.
- Use SQLite’s backup API to copy the synthetic database into a fresh destination. Re-enable connection-level foreign-key enforcement as required and compare scoped orders, line values, stock and replay records.
- Run integrity and foreign-key checks for their distinct purposes. An integrity result is not a substitute for checking reference violations or business totals.
- Attempt cross-tenant reads/writes and a forbidden action. Assert that outputs and stored state respect the trusted-helper contract. Then explicitly note that unrestricted SQL access lies outside that contract.
51.5.4 PLAIN — what is really happening inside#
- Rollback restores the transaction’s database changes. It cannot undo an external side effect already performed, which is why the lab records intent instead of dispatching an email inside the transaction.
- A backup API coordinates copying according to the engine’s rules. The copied database still needs validation and appropriate application configuration before it is used.
- The test harness supplies known state and forced interruption points. It checks the resulting state against an expected contract rather than inferring success from an exception name alone.
51.5.5 TECHNICAL — the engineer’s version#
- Python’s sqlite3 API exposes backup and explicit transaction management; SQLite documents the distinct integrity and foreign-key checks. The final test record identifies the actual Python/SQLite runtime used. [S55] [S59] [S86]
- The earlier two-connection temporary-file locking exercise demonstrates one bounded real SQLite writer-contention scenario. The capstone’s deterministic models and in-memory tests do not expand that into a multi-host concurrency claim.
- Production evidence would additionally need genuine authentication/role integration, representative concurrent workloads, restore objectives, failure-domain tests, secret handling and lifecycle verification across actual copies. Those are a deployment programme, not an unreported result of this lab.
51.5.6 WORDS — remember these#
Fault injection: deliberately interrupting a known path — a test technique whose simulated failure must be named precisely. Restore rehearsal: recovering and checking a separate destination — evidence about a specific recovery procedure and input. Integrity check: inspection of declared structural properties — one layer of validation, distinct from complete business or security correctness.
51.6 Review and release evidence#
51.6.1 PLAIN — in simple words#
- A finished teaching package should let another reader see which examples were implemented, which tests ran and which discussions remain conceptual.
- Keep manuscript, code and evidence aligned. A paragraph claiming a particular lab exists is misleading if the download contains no such exercise.
- Test counts measure cases executed, not overall truth. A test can correctly demonstrate missing protection, and a large suite can still omit an important requirement.
- A public learning book is not a production-service certificate. Readers should be able to learn from the evidence without inheriting an unsupported guarantee.
51.6.2 PLAIN — a picture in your head#
- Mira hands Dev a recipe, the ingredients actually used and the notes from a trial batch. He can distinguish what was cooked from what was only proposed.
- If the trial used one small oven, the notes should not claim that a factory production line was tested.
- Where the comparison breaks: reproducibility also depends on software versions and exact input bytes. A familiar filename alone may not identify the same candidate.
51.6.3 PLAIN — a worked example#
- Extract the companion code into a fresh folder and read its README before execution. Run the supplied unittest discovery command with the supported Python version. The output identifies successes, failures and any explicit skips.
- Read the lab inventory. The foundations modules cover Chapters 1–12; advanced_lab contains bounded models and query/representation exercises for later topics; capstone_lab assembles the scoped local order workflow and Part H calculations.
- Compare the included QA report with the package manifest. The report identifies the actual test run and generated artefacts; the manifest supports checking whether a later copy has changed.
- A skipped optional capability is not a passed test. An illustrative PostgreSQL statement remains documented rather than executed unless the evidence explicitly identifies a real PostgreSQL run.
- The package’s website is the book reader, not a deployed checkout application. Publishing the reader makes educational text and downloads available; it does not install the capstone as a commercial service.
51.6.4 PLAIN — what is really happening inside#
- A test suite supplies executable assertions over known inputs. A build pipeline converts one manuscript into PDF, volume and browser representations. Structural checks then compare chapter coverage, identifiers and resource targets.
- Visual inspection catches a different class of problem: a clipped formula, unreadable table or misplaced heading may survive a text-only check.
- The release record should distinguish automatic checks, visual review, source research and any independent review. Do not invent a reviewer or declare that one authoring pass is independent of itself.
51.6.5 TECHNICAL — the engineer’s version#
- Python unittest’s result model distinguishes failures, errors and skips. The supplied runner records the runtime and actual counts rather than hardcoding a flattering number in the manuscript. [S201]
- Stable chapter and section identifiers support cross-edition reference even when a standalone volume has different page numbers. Each volume’s contents refer to that volume; its chapter numbers match the complete edition.
- A package manifest is an integrity aid, not authentication of its publisher by itself. A trustworthy distribution channel and an independently obtained expected digest are needed for stronger origin claims.
- The practical conclusion is not “never trust a system.” It is to define the required claim, inspect the mechanism, run a suitable observation and keep the conclusion within the evidence.
51.6.6 WORDS — remember these#
Executable assertion: a machine-checkable expectation — a predicate comparing an observed result with a declared outcome. Release manifest: an inventory of delivered bytes — filenames, sizes and digests that support integrity checks. Evidence boundary: the limit of a justified conclusion — the tested scope, assumptions and unresolved properties attached to a result.
51.97 Practice and worked answers#
- Question: What stock remains after CAP-001 and its identical retry? Answer: Eight notebooks and forty-nine pens in T-A/B-1; the retry returns the accepted result without a second deduction.
- Question: What happens if R-1 returns with a different quantity? Answer: A conflicting command under the same scoped identity is refused; the old result is not overwritten.
- Question: Does repricing NOTE change an accepted CAP-001? Answer: No. Its line prices remain the accepted snapshot.
- Question: What does two deliveries and one task establish? Answer: The tested local inbox/task protocol suppresses an identical repeated delivery. It does not establish exactly-once external dispatch.
- Question: Why compare both tenants’ state after a denied write? Answer: A denied response alone does not prove that no protected state changed.
- Question: Does this capstone explain the original stock mismatch? Answer: No. Its new fixtures are separate; the original expected-ten/observed-nine discrepancy still has no asserted cause.
- Question: What does the backup test not establish? Answer: Off-site recovery, device power-loss durability, PostgreSQL recovery and production objectives remain outside its scope.
- Question: Is the website reader a production checkout deployment? Answer: No. It publishes educational content and downloads, not a live order-processing service.
51.98 Common wrong ideas#
- Wrong: A capstone must implement every commercial requirement. Right: A useful educational system has explicit included claims and exclusions.
- Wrong: A request key alone creates idempotency. Right: Scope, payload comparison, atomic effects and retention also matter.
- Wrong: The current catalogue is the historical agreement. Right: Accepted line prices are separate facts.
- Wrong: An outbox guarantees exactly one external action. Right: It records intent; receiver and external-effect boundaries remain.
- Wrong: A cross-tenant SELECT test proves all access paths are safe. Right: Exports, workers, mutations and privileged interfaces need their own coverage.
- Wrong: Matching totals reconcile everything. Right: Compare scoped identities and values as well.
- Wrong: Exception injection tests physical durability. Right: It exercises only the injected software path and transaction outcome.
- Wrong: Many passing tests make the publication infallible. Right: Test coverage and factual/editorial review remain distinct forms of evidence.
51.99 Chapter summary in 20 lines#
- Begin the capstone with explicit requirements and exclusions.
- Keep new fixtures separate from the original canonical records.
- Establish principal and tenant scope before protected operations.
- Validate bounded command shapes and integer quantities.
- Capture agreed prices separately from mutable catalogue prices.
- Give every table a clear row grain and scoped identity.
- Own one transaction for each accepted local operation.
- Keep stock, order, request result and outbox intent atomic.
- Compare repeated payloads under the same request identity.
- Recover a lost response without creating another order.
- Commit consumer inbox and local task together.
- Distinguish redelivery from repeated local effect.
- Do not extend local deduplication to uncontrolled external actions.
- Reconcile keys, quantities, prices and scoped totals.
- Test refused operations and their stored effects.
- Inject bounded failures and inspect rollback.
- Restore into a fresh destination and check useful records.
- Identify the actual runtime, assertions and skips.
- Keep text, code, downloads and release records consistent.
- Carry the evidence discipline from one shop to every larger system.