work / research platforms
QTS research data platform
The market-data platform I architected as sole developer for the CUHK Quant Trading Society's research team. Version-pinned reads let researchers rerun a study on the same rows.
the engineering
The research team needed tick-level A-share and crypto data they could trust and reproduce later on exactly the same rows. It arrived as compressed vendor archives, each with its own quirks.
- Raw deliveries are kept byte-identical; nothing upstream is edited in place.
- Every read is pinned to a data version and may never see the future.
- One developer, so every cleansing rule has to stay visible and auditable.
The result, computed
BUILD-COMPUTED by the Lab's own code when this page was built · a SYNTHETIC stock whose corrections arrive on later days
| field | read as of Mon | read as of Sat | versions hidden on Mon |
|---|---|---|---|
| close (Mon) | 10 | 10.4 | 1 |
| volume (Mon) | 1200 | 1350 | 1 |
| split factor | 1 | 2 | 1 |
| close (Tue) | not known yet | 10.9 | 1 |
| dividend (Wed) | not known yet | 0.3 | 1 |
A study run on Mon sees only what was known on Mon. Rerun it later, pinned to the same version, and it gets exactly the same rows; the corrections stay out of the past.
Try it livemove the read date and watch which versions become visibleWhat it shows
A read as of Monday sees only what was known on Monday; a correction published later appears only for later reads. The platform's separate 420,000-row concurrency fixture kept all 23 of 23 commits with 12 readers and 5 writers. That checks versioning under contention, not production-scale throughput.
The tradeoff
- chose
- a query interface with pinned versions, explicit errors and no look-ahead
- over
- hand-written DuckDB SQL straight on the files
- because
- a study rerun later must return exactly the same rows, and a look-ahead leak must be impossible, not merely unlikely
- cost
- it is deliberately slower than raw SQL; raw DuckDB stays available for anyone who wants the speed
What I built
As the team's sole developer: the layered store, the versioned query service and shared research infrastructure. The team originated the data-quality work; I led its implementation with them.
- 700B+
- stored A-share and crypto rows in the 2026-10-05 snapshot; not a unique-row count
- 23 of 23
- commits kept on a separate 420,000-row concurrency fixture with 12 readers and 5 writers
The problem
The research team needed tick-level Chinese A-share and crypto data that they could trust and reproduce later on exactly the same rows. The data arrived as compressed vendor archives, each with its own quirks, and no one on the team was building the infrastructure full time. As the only developer, I owned the whole stack.
What I built
- A layered store. Raw deliveries are kept byte-identical. A typed, versioned Parquet layerversioned ParquetColumnar data files kept as immutable versions, so a query can name the exact version it read and get the same rows again. sits above them, and a cleansed layer above that flags bad rows instead of dropping them.
- A query service for researchers in Python over HTTP and SQL. It returns Apache Arrow, pins every read to a data version, and refuses requests that would see the futurelook-ahead biasUsing information in a backtest that was not yet available when the decision was made. It makes results look better than anything that could have traded..
- Data cleaning, led with the team: exact duplicates removed, look-ahead leaks blocked before they reach research data.
- Shared research infrastructure: access controls in C, monitoring, transition-based alerting and operator documentation.
- A market-data relay in Go that validates requests, terminates TLS and scrubs addresses for HTTP and WebSocket traffic. It replaced a Python prototype; the recorded acceptance walkthrough checked collector connections and remote-login continuity.
- Guarded AI coding agents: recoverable deletion and approvals, checked against independently identified bypass cases. Those cases do not establish complete security coverage.
Result
- 700B+ stored rows in the 2026-10-05 snapshot: A-share ticks plus crypto market data. The platform also supports Polymarket, which is outside this sum.
- A separate 420,000-row concurrency fixture: 12 readers and 5 writers (4 appending, 1 rewriting a partition) kept all 23 of 23 commits; 1,111 version-pinned reads returned without mismatch.
- 500M+ exact A-share duplicates removed, recorded in the same snapshot, from the cleaning work I led with the team.
How it works
- 01Vendor archivescompressed, quirky
- 02Raw layerbyte-identical
- 03Typed Parquetversioned
- 04Cleansed layerflag, never drop
- 05Query serviceArrow, pinned, as-of
Three decisions shaped it:
- Flag, never drop. Only exact duplicates are deleted. A cleansing rule that silently removes rows is a rule nobody can audit.
- Three layers, not two. When a cleansing rule changes, it reruns from the typed layer instead of re-decoding archives, and every rule’s effect stays visible.
- Safety over peak speed. The query interface is deliberately slower than hand-written DuckDB SQL. In exchange it gives pinned versions, explicit errors and no look-ahead. Raw DuckDB is still there for those who want it.
Limits
- The platform’s code and data are closed, so the figures above cannot be reproduced from a public repository. The write-up explains the design, and the demo shows the same guarantees on synthetic data.
- The write-up reports no uptime or researcher-usage figures.
- The snapshot counts stored rows, not unique observations. The concurrency fixture is a correctness check on a small dataset.
- The first query on a new version rebuilds its file list and was not timed. No cold-query latency bound is claimed here.