study_id_patterns.py — Study-ID Pattern Registry
BCHPR · 9 projects, 4 countries · 2023 – present
The most-imported module in the codebase — 35 files depend on it. 27 registered patterns (16 participant and specimen ID forms, 11 site codes) covering identifiers that arrive from QR scanners, free-text notes and typed entry, where a transposed digit produces something structurally valid and completely wrong.
Highlights
- A vectorised ISO 7064 Mod 37,36 check-character kernel that reads a UCS-4 string array as a raw uint32 buffer and iterates character positions rather than payloads — roughly 30× the scalar path at 100k IDs, and caching cannot help because participant IDs are near-unique.
- One combined alternation regex replaces per-pattern looping: 100,000 rows went from 1.2 million regex operations to 100,000.
- A rule narrowed after a real incident — the DARE-TB site digit accepted [1-9] until July 2026, which admitted IDs no site could mint; QA found two of them sitting in a live REDCap project.
- Validation returns typed errors (invalid format, wrong project, wrong country, failed check character, duplicate) with a suggested correction, not a boolean.
- Cleaning handles the specific corruptions that occur: OCR O-to-zero confusion, missing hyphens, trailing scanner noise, and IDs concatenated with no separator.
- Garbage collection gated behind a 100,000-row threshold after 67 frames of a few rows each paid 3.2 seconds of pure GC on a memory-constrained WSL run.
Related projects
Data Manager
data_quality_manager.py — Data-Quality Engine
REDCap validates one field at a time, at entry. It cannot express “this specimen date precedes the enrolment date on a different form”, or “these two record IDs are the same participant”. This 14,274-line engine does, and it remembers — every issue it raises has an open date, a close date, and a resolution time.
Data Manager
MANAGER.py — Manager.io Accounting API Client
15,661-line Manager.io API client. Full create / update / delete across 18 document types — sales quotes, orders, invoices, credit notes, delivery notes and receipts; purchase quotes, orders, invoices and debit notes; inventory items, locations, transfers, write-offs, goods receipts, production orders and starting balances; and payments. Customers, suppliers, projects, accounts, assets and tax codes are exposed read-only.
Data Manager
data_utilities.py — Shared DataFrame Toolbox
The dependency root of the library — 9,746 lines that import nothing else in the codebase and that everything else imports. 84 public callables covering REDCap activity-log parsing, data-dictionary recoding, checkbox column expansion, export diffing, and file I/O that survives a SharePoint sync.