ToddleAI

What It Does

You record a short, guided video of your toddler walking. ToddleAI checks the recording quality in real time as you shoot — if the feet aren't visible, the camera is shaking, or your child isn't walking side-on, it tells you exactly what to fix before you even stop recording. Once you have a good clip, it extracts step-by-step timing from the walk (step time, cadence, left/right symmetry) and compares them to published pediatric norms. You can then ask an on-device assistant to explain the results in plain language. Every step — video, pose estimation, gait math, and chat — runs locally on the phone; the app requests no internet permission.

Key Features

Guided Capture. CaptureScreen overlays a live MediaPipe skeleton, a dashed framing guide-zone, an animated corner/arrow prompt (FramingArrowOverlay) telling you to move back, tilt down, pan left/right, etc., and a step counter that pops as steps are detected. GuidanceEngine drives a running coaching message ("3 good steps — 2 more needed", "Great capture!") based on a rolling window of per-frame quality.

Quality Gate. Rejects bad recordings and tells the parent exactly why — see its own section below. This is ToddleAI's signature feature.

Temporal Gait Metrics. Computed per clip: cadence (steps/min, from median step time), left/right step-time symmetry (mean timing difference and asymmetry %), and step-time variability (coefficient of variation, a rhythm-consistency measure). All are temporal only — no stride length, joint angles, or walking speed are computed.

On-Device LLM Agent. A Llama 3.2 1B Instruct model, run through ExecuTorch's LlmModule, answers parent questions grounded in that clip's actual observations (injected via PromptBuilder.gaitContext). It is a single grounded chat completion today, not a multi-tool agent — LlamaAgent.kt and ToolRouter.kt are present as stubs for a planned tool-calling layer but contain no implementation yet.

Privacy by architecture. The manifest requests only CAMERA — no INTERNET permission exists anywhere in the app, so no gait data, video, or chat text can leave the device even if the code tried. The Settings screen verifies and displays this at runtime.

Architecture

In-app path (what actually ships in the APK): pose estimation runs through MediaPipe Tasks PoseLandmarker, backed by TFLite + the XNNPACK CPU delegate (a GPU delegate is available but CPU is default). The on-device LLM chat runs through ExecuTorch's XNNPACK/CPU backend as well — see NPU / ExecuTorch Integration for why QNN isn't used for either workload in the shipping app.

Adb-shell NPU benchmark path (separate from the app): samples/pose/ and runqwennpu.sh drive Qualcomm-exported .pte models through qnnexecutorrunner / qnnllamarunner directly over adb shell, exercising the Hexagon NPU outside the installed app's sandbox. This is the dual-path design: one path is what a parent's phone actually runs; the other is the NPU capability proof that the installed app itself cannot reach (see below).

Two-pass design: during recording, FrameProcessor runs a lightweight per-frame quality check and feeds GuidanceEngine/FramingGuide for live coaching text and overlay color — no gait metrics are computed live. After recording (or after importing a clip), ReplayAnalyzer re-runs event detection twice: once on the raw buffered frames to pick the best contiguous "good" segment, then again on a 5-frame-smoothed, landmark-filtered version of that segment to compute the final metrics. Imported videos go through the exact same ReplayAnalyzer as live recordings.