andydataguy
AI & SYSTEMS . AI and Data Solutions . 2024

Algorithmic Trading and Data Engineering

When Mistakes Cost Houses

Andy Houston author chipBY ANDY HOUSTON
Real-time data ingestion through validation gates with sub-ms latency counterEQUITY FEEDVALIDATEFILTERFX FEEDVALIDATEFILTERCRYPTO FEEDVALIDATEFILTEROPTIONS FEEDVALIDATEFILTERWARNP95 LATENCY0.84 MS
Milliseconds
decision windows
Stable
trading behavior
Visible
failure modes
Boring
reliability

Context

Crypto trading firm whose systems ingested large volumes of noisy market data and made real-money decisions in milliseconds. Crypto never closes, so the feeds never stop and there is no overnight window to catch a bad assumption before it trades. I came in from high-budget, high-volume advertising data work, aggregating from multiple sources and running attribution. Trading data was more intensive than all of it.

Problem

Data quality and latency issues threatened both performance and risk controls. A single bad assumption could translate into outsized financial loss. Bad data in a system like this costs money directly: lost trades, missed opportunities, and systematic errors that compound quietly until someone goes looking. A bug did not just cost a report. It could cost a house.

Approach

Helped design and refine ingestion pipelines, data validation layers, and monitoring. Built technical and custom indicators, matched them to different market regimes, accounted for different asset types, and cross-referenced all of it with prediction pipelines. On top of that sat a layer of agents reading news, market microstructure, and confidence scoring in real time, running on Supabase and LangGraph. Focused on separating true signal from noise, making sure models were fed trustworthy inputs and that failure modes were visible before they became catastrophic.

Signal-vs-noise filtering through the validation pipeline: raw feed in, filtered signal out.

SIGNAL vs NOISE . VALIDATION PIPELINERAWFILTEREDINGESTVALIDATEFILTERPREDICTMONITOR

Stack

  • Python
  • SQL
  • Supabase
  • LangGraph
  • Real-time data feeds
  • Technical and custom indicators
  • Prediction pipelines
  • Logging and monitoring stacks

Result

Milliseconds
decision windows

More stable trading behavior, clearer insight into when and why models performed the way they did, and a stronger foundation for further experimentation. A $60K build, and around $35K of that survived as profit after hard expenses.

MILLISECONDS · DECISION WINDOWS

Millisecond decision-window trace with stability and failure-mode indicators.

DECISION WINDOWSREAL-MONEY DECISION WINDOWMillisecondsTRADING BEHAVIORSTABLEFAILURE MODESVISIBLE

Impact

Raised the floor on risk. Operators could finally reason about the system instead of trusting it. It is still the project I name first when someone asks what I built in 2024.

Lessons

In high-stakes environments, boring reliability beats cleverness. You earn the right to innovate by making the basics unbreakable. You will never have perfect information in a market, so the question that matters is whether there is enough signal to act at acceptable risk, and how fast you can answer it. The engagement brought me to an entirely different level of data engineering.

Why this matters to you

For any team where data quality, latency, and reliability directly touch revenue or safety.

Sounds like you? Get a quote