Linkt Time Use Diary
Company: Datacubed Health, 2017-2018
Role: Chief Technology Officer
Team: UX Designer, Visual Designer, Data Scientist, Research Scientist
Problem
Y-Combinator Research was looking to conduct a large scale study of basic income and its impact on people's lives. The study would measure the daily behaviors of 1000 people over a six month period with some of the participants receiving $1000 in supplemental income.
To effectively measure this, YCR needed a tool which could truly reassemble the participant’s day to day habits and pattern changes over 1-2 year period.
The typical paper survey for measuring this information takes a participant anywhere from 8-15 minutes to complete, not including time for the researcher to code and input that information into a digital system. YCR worked with our team to help develop a new solution.
Key questions we wanted to answer:
Will basic income decrease time spent working?
Will quality of life metrics such as time spent with family or friends increase?
Will behavioral changes be significant enough to measure?
Challenges
Paper time use surveys require the user to recall the previous day’s events, which is subject to high amounts of bias and cognitive burden.
Machine learning could have two relevant uses, segmenting the users day based on identified patterns in the data sets as well as prioritizing the most relevant interface choices based on data patterns and previous entries.
Training would be unique for each users patterns, but this diary was required to be filled out 12-18 times, which meant improvements would reduce the overall diary burden over time and could be measured throughout the process.
Solution
Our Linkt mobile app collects raw data such as location, phone usage, fitness tracker and sleep sensor information, but for this project we would need to break down time usage in 15 minute segments to allow users to provide their day to day activities up to 18 times during the study (up to 4.5 hours over the course of the full study!)
I led our team through a 6 month development and research activity to design a time use diary that could intelligently segment time use using patterns identified using supervised (classification) and unsupervised (cluster analysis) machine learning algorithms on top an existing user interface.
The time use diary UX allowed the user to log their daily activities using an interaction pattern similar to mobile calendar apps.
My role included managing the design team, creating product requirements, planning the roadmap, and supporting iterative decision making as we learned what data sets would and would not properly give us useful information.
I managed our team data scientist who worked closely with our research scientist and built out many of the test models in Jupyter using Python (SciPy/NumPy). Working with our production engineers, this information was returned to the device through an API containing the users most relevant choices and time segments based on the data they logged in the study up to that point.
An example workflow:
In this specific example, we reduce the amount of selection options for any given entry by using known data points to limit unnecessary options. For example, we would never surface questions about caring for a child if the participant indicated they had no children during their onboarding.
This approach had an added benefit that over time, we would be able to assume information about partial days when a user might not actually complete their diary.
Changes over the course of the year long period with and without basic income would be able to detect meaningful shifts in behavior.
Key data points and learnings from the development process
Latitude and longitude: Using location cluster analysis we could calculate dwell time, time in transit, home, school and work locations.
Bonus: For NYC residents we were able to calculate the method of transit (subway, bus, bicycle, walking) by cross referencing locations of known bus stops and subway stations and average velocity.
By using the selected answers from the superset of all users and individual users we were able to make determinations of more specific events with a high (80%+) accuracy.
Time in transit between home and work was able to be predicted with 95%+ accuracy using our models. However, this model would break for users who had multiple jobs or worked from home.
Sleep time was only feasible to predict for about 75% of our users based on phone usage activity. Users who did not typically use their phone while they were at home would almost always have over-estimated sleep durations which required manual correction. With additional development, this could have been improved by using the survey responses or sensor data to estimate their usual sleep patterns.
Measuring Success
Time to Task completion
Average Paper Time Use Survey Completion time: 8m30s*
Initial Mobile App Completion Time: 4m36s (Excluding tutorial)
Second Pass (No Improvement Algorithms): 2m15s
Second Pass with ML optimizations: 1m58s
Third-Fifth Pass with optimizations: 1m26s
This approach with the UX only represented a massive reduction in the overall time necessary, however the addition of using user supervised and passive data to train improvements represented over a 35% improvement in the time of the diary.
*Limited sample size, US Dept of Labor and experts quote 15-20 minutes
Project Outcomes
The team was extremely happy with our overall result, and the key researchers on the project went on to present these findings at a time use summit (yes there is such a thing). The overall project however was put on hold after the costs to conduct the study caused substantial delays and hampered their ability to fund the long term research.
The biggest outcome the project aimed to advance was the ability to use this type of data to predict behavior changes based purely on passive data with less requirement on the manual data entry. That type of project would allow a massive data set to help predict large scale behavior change.