Role: Creative Lead & UX Engineer

“
I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”
John Gasper
Sr. UX Researcher at Google
From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.
The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.
Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?
Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.
As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.
What does this unlock?

01
Richer signals for routines
Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.
02
Whole home automation
Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.
What we produced

Zone Framework
We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit
We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases
We applied the zone framework to a high-value use case: a multi-person household going to bed.
Zone Framework
HOUSEHOLD ZONES
A mental model of household zones
Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS
Sequences
Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.
Rhythms
In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit
How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?
STUDIES
Our Cuboro kit connected multiple soli devices to identify patterns
By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.
ZONES
Spatial density maps help identify zones
Initial experiments tracked repeated coffee making routines over the course of a month. Using captured data, spatial density maps were developed to visualize the duration of time spent in each zone of the tracking area. For each spatial density map created, areas of significance can be identified by warmer colors. Mid tones act as a good indications of commonly used pathways between hotspots.

We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.
Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES
Movement between zones over time help identify sequences
Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.
This bar shows the sequence of a coffee routine.
RHYTHMS
Several Soli devices help identify the rhythms of a home
With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.
As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.
Use Case: Sleep
We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME
Multi-soli for whole-home sleep intelligence
With Soli in a growing number of Google Nest products, there is an opportunity to expand our understanding of spatial context at a more granular level than previously possible. The simple understanding of device type and location can help build richer contextual understanding and prediction that we can leverage with ARCHIE, Nest’s context engine.

Sleep rhythmsextend beyond the bedroom

Entertainment zones
Public zones for entertainment and relaxation. Device: Television

Transient zones
Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones
Pivotal zones for rest and sleep in the home Device: Hub
USE CASE 1
First person goes to bed
In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.
USE CASE 2
Last person goes to bed
Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.
By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.
These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.
Experiences
What we unlock with zones

Security
First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.
Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting
First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.
Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment
First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.
Last person to bed: Any devices still on are switched off, including the paused living room TV.
State of the home
What can happen when both people are asleep

Entertainment
When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant
Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy
Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.
The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.
Contributions
Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation
Collaborators
ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),
Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)
Patent
System of Multiple Radar-Based Gesture Detectors, 2022
Role: Creative Lead & UX Engineer

“
I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”
John Gasper
Sr. UX Researcher at Google
From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.
The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.
Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?
Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.
As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.
What does this unlock?

01
Richer signals for routines
Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.
02
Whole home automation
Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.
What we produced

Zone Framework
We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit
We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases
We applied the zone framework to a high-value use case: a multi-person household going to bed.
Zone Framework
HOUSEHOLD ZONES
A mental model of household zones
Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS
Sequences
Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.
Rhythms
In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit
How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?
STUDIES
Our Cuboro kit connected multiple Soli devices to identify patterns
By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.
ZONES
Spatial density maps help identify zones
Initial experiments tracked repeated coffee making routines over the course of a month. Using captured data, spatial density maps were developed to visualize the duration of time spent in each zone of the tracking area. For each spatial density map created, areas of significance can be identified by warmer colors. Mid tones act as a good indications of commonly used pathways between hotspots.

We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.
Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES
Movement between zones over time help identify sequences
Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.
This bar shows the sequence of a coffee routine.
RHYTHMS
Several Soli devices help identify the rhythms of a home
With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.
As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.
Use Case: Sleep
We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME
Multi-soli for whole-home sleep intelligence
With Soli in a growing number of Google Nest products, there is an opportunity to expand our understanding of spatial context at a more granular level than previously possible. The simple understanding of device type and location can help build richer contextual understanding and prediction that we can leverage with ARCHIE, Nest’s context engine.

Sleep rhythms extend beyond the bedroom

Entertainment zones
Public zones for entertainment and relaxation. Device: Television

Transient zones
Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones
Pivotal zones for rest and sleep in the home Device: Hub
USE CASE 1
First person goes to bed
In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.
USE CASE 2
Last person goes to bed
Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.
By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.
These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.
Experiences
What we unlock with zones

Security
First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.
Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting
First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.
Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment
First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.
Last person to bed: Any devices still on are switched off, including the paused living room TV.
State of the home
What can happen when both people are asleep

Entertainment
When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant
Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy
Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.
The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.
Contributions
Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation
Collaborators
ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),
Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)
Patent
System of Multiple Radar-Based Gesture Detectors, 2022
Role: Creative Lead & UX Engineer

“
I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”
John Gasper
Sr. UX Researcher at Google
From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.
The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.
Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?
Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.
As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.
What does this unlock?

01
Richer signals for routines
Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.
02
Whole home automation
Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.
What we produced

Zone Framework
We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit
We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases
We applied the zone framework to a high-value use case: a multi-person household going to bed.
Zone Framework
HOUSEHOLD ZONES
A mental model of household zones
Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS
Sequences
Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.
Rhythms
In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit
How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?
STUDIES
Our Cuboro kit connected multiple soli devices to identify patterns
By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.

ZONES
Spatial density maps help identify zones
Our first experiments tracked a repeated coffee-making routine over a month. From the data, we built spatial density maps showing how long people spent in each zone of the tracking area. Warmer colors mark the areas that matter; mid tones show the paths people commonly use between hotspots.
We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.
Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES
Movement between zones over time help identify sequences
Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.
This bar shows the sequence of a coffee routine.
RHYTHMS
Several Soli devices help identify the rhythms of a home
With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.
As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.
Use Case: Sleep
We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME
Multi-soli for whole-home sleep intelligence
With Soli in a growing number of Google Nest products, we could understand spatial context in more detail than before. Knowing each device’s type and location helps build richer context and predictions for Nest’s context engine.

Sleep rhythms extend beyond the bedroom

Entertainment zones
Public zones for entertainment and relaxation. Device: Television

Transient zones
Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones
Pivotal zones for rest and sleep in the home Device: Hub
USE CASE 1
First person goes to bed
In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.
USE CASE 2
Last person goes to bed
Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.
By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.
These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.
Experiences
What we unlock with zones

Security
First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.
Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting
First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.
Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment
First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.
Last person to bed: Any devices still on are switched off, including the paused living room TV.
State of the home
What can happen when both people are asleep

Entertainment
When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant
Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy
Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.
The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.
Contributions
Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation
Collaborators
ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),
Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)
Patent
System of Multiple Radar-Based Gesture Detectors, 2022