From Records to Insight
Build pipelines, analytical systems and search while preserving the meaning of the source data.
Data Pipelines: Capture, Transform and Deliver
A data pipeline carries information from one representation or system into another. It may copy, validate, reshape or summarise the records, but it must…
4,030 words/18 min read41Streams, Event Time and Late Arrivals
A stream keeps bringing records while a question is being answered. There may be no natural final record telling the system that a day’s information is…
3,919 words/17 min read42Warehouses, Lakes and Analytical Tables
The database that accepts one customer’s order and the system that studies five years of orders solve different workloads. They may share data, but they…
3,873 words/17 min read43Columnar Storage and Compression
A report often needs a few fields from many records. Columnar storage arranges values so the engine can avoid reading unrelated fields and can exploit…
3,861 words/17 min read44Full-Text Search and Relevance
Looking up an exact order ID and finding a useful document are different tasks. Text search must decide how to split language, which expressions match…
3,773 words/16 min read45Vector Search: Similarity Is Not Truth
Vector search represents items as lists of numbers and retrieves items that are close under a chosen mathematical rule. It can find useful similarities…
3,807 words/17 min read46Metrics, Bias and Honest Analysis
A metric is a precisely defined question expressed as a number. The calculation can be flawless while the definition, population or interpretation is…
3,691 words/16 min read