Beyond
Reads heart rate, HRV and blinks from an ordinary webcam, in real time.
- Year
- 2022 – 2026
- Role
- Lead developer
- Stack
- Python · OpenCV · dlib · FastAPI · WebSockets · React
The problem
In a video consultation, a clinician loses most of the physical signals they would read in person. Beyond grew out of my MSc research on augmented teleconsultation for depressive disorders: it extracts physiological signals from the video feed so they can support the clinician’s assessment.
How it works
- Each frame goes through face detection and 68-point facial landmarks.
- Remote photoplethysmography (rPPG) recovers the pulse from tiny colour changes in the skin, giving heart rate and HRV metrics (SDNN, RMSSD, LF/HF).
- The eye aspect ratio detects blinks, and landmark velocity tracks head movement.
- A Python SDK exposes one Analyzer class, with a FastAPI WebSocket server and a React frontend for live sessions.
Key decisions
One small public API
Everything sits behind Analyzer.process_frame(), returning typed results with a confidence score, so the SDK drops into any video pipeline, including a telemedicine platform.
Signals clinicians already use
Heart rate variability, blink rate and movement are established markers in mental-health research, which kept the output interpretable rather than a black-box score.
Results
- Published in IEEE Access (Q1), in collaboration with Hospital Beatriz Ângelo
- Best Master Thesis in Innovation, Health Systems and Digital Transformation, ULisboa 2022
- Explored commercialisation with VCs and the Portuguese NHS
Each dip in the eye aspect ratio is a detected blink; the pulse signal and its peaks run underneath
What I’d do next
Validating accuracy across skin tones and lighting conditions, and moving inference into the browser.
Built with Pedro Constantino and Vasco Duarte.