Google

Zones in the Home

Role: Creative Lead & UX Engineer

“

I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”

John Gasper

Sr. UX Researcher at Google

From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.

The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.

Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?

Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.

As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.

What does this unlock?

01

Richer signals for routines

Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.

02

Whole home automation

Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.

What we produced

Zone Framework

We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit

We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases

We applied the zone framework to a high-value use case: a multi-person household going to bed.

Zone Framework

HOUSEHOLD ZONES

A mental model of household zones

Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS

Sequences

Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.

Rhythms

In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit

How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?

STUDIES

Our Cuboro kit connected multiple soli devices to identify patterns

By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.

ZONES

Spatial density maps help identify zones

Initial experiments tracked repeated coffee making routines over the course of a month. Using captured data, spatial density maps were developed to visualize the duration of time spent in each zone of the tracking area. For each spatial density map created, areas of significance can be identified by warmer colors. Mid tones act as a good indications of commonly used pathways between hotspots.

We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.

Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES

Movement between zones over time help identify sequences

Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.

This bar shows the sequence of a coffee routine.

RHYTHMS

Several Soli devices help identify the rhythms of a home

With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.

As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.

Use Case: Sleep

We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME

Multi-soli for whole-home sleep intelligence

With Soli in a growing number of Google Nest products, there is an opportunity to expand our understanding of spatial context at a more granular level than previously possible. The simple understanding of device type and location can help build richer contextual understanding and prediction that we can leverage with ARCHIE, Nest’s context engine.

Sleep rhythmsextend beyond the bedroom

Entertainment zones

Public zones for entertainment and relaxation. Device: Television

Transient zones

Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones

Pivotal zones for rest and sleep in the home Device: Hub

USE CASE 1

First person goes to bed

In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.

USE CASE 2

Last person goes to bed

Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.

By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.

These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.

Experiences

What we unlock with zones

Security

First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.

Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting

First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.

Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment

First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.

Last person to bed: Any devices still on are switched off, including the paused living room TV.

State of the home

What can happen when both people are asleep

Entertainment

When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant

Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy

Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.

The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.

Contributions

Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation

Collaborators

ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),

Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)

Patent

System of Multiple Radar-Based Gesture Detectors, 2022

Google

Zones in the Home

Role: Creative Lead & UX Engineer

“

I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”

John Gasper

Sr. UX Researcher at Google

From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.

The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.

Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?

Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.

As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.

What does this unlock?

01

Richer signals for routines

Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.

02

Whole home automation

Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.

What we produced

Zone Framework

We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit

We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases

We applied the zone framework to a high-value use case: a multi-person household going to bed.

Zone Framework

HOUSEHOLD ZONES

A mental model of household zones

Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS

Sequences

Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.

Rhythms

In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit

How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?

STUDIES

Our Cuboro kit connected multiple Soli devices to identify patterns

By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.

ZONES

Spatial density maps help identify zones

Initial experiments tracked repeated coffee making routines over the course of a month. Using captured data, spatial density maps were developed to visualize the duration of time spent in each zone of the tracking area. For each spatial density map created, areas of significance can be identified by warmer colors. Mid tones act as a good indications of commonly used pathways between hotspots.

We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.

Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES

Movement between zones over time help identify sequences

Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.

This bar shows the sequence of a coffee routine.

RHYTHMS

Several Soli devices help identify the rhythms of a home

With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.

As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.

Use Case: Sleep

We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME

Multi-soli for whole-home sleep intelligence

With Soli in a growing number of Google Nest products, there is an opportunity to expand our understanding of spatial context at a more granular level than previously possible. The simple understanding of device type and location can help build richer contextual understanding and prediction that we can leverage with ARCHIE, Nest’s context engine.

Sleep rhythms extend beyond the bedroom

Entertainment zones

Public zones for entertainment and relaxation. Device: Television

Transient zones

Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones

Pivotal zones for rest and sleep in the home Device: Hub

USE CASE 1

First person goes to bed

In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.

USE CASE 2

Last person goes to bed

Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.

By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.

These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.

Experiences

What we unlock with zones

Security

First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.

Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting

First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.

Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment

First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.

Last person to bed: Any devices still on are switched off, including the paused living room TV.

State of the home

What can happen when both people are asleep

Entertainment

When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant

Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy

Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.

The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.

Contributions

Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation

Collaborators

ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),

Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)

Patent

System of Multiple Radar-Based Gesture Detectors, 2022

Google

Zones in the Home

Role: Creative Lead & UX Engineer

“

I just came across your Zones in the Home work in the upcoming Ambient newsletter and it's definitely one of the coolest things I've seen done in the ambient computing space so far.”

John Gasper

Sr. UX Researcher at Google

From September 2020 to March 2021, I led a research and development initiative at Google ATAP on Soli radar sensing for presence in the home.

The ambient computing paradigm shifted the focus from personal devices to collections of communal devices that operate in the background as part of one single computer. However, we have limited techniques to coordinate device behavior, and make them respond in concert to people's everyday activities, without their intervention.

Soli is a sensing technology that uses miniature radar to understand movement at different scales: from the twist of a finger to the turn of your body.

Multi-Soli explored the question: How can we combine multiple Soli in one product or across multiple products to create new experiences?

Our hypothesis was that by identifying zones in the home, we could detect everyday routines and coordinate a whole collection of devices around them.

As Creative Lead and UX Engineer, I set the creative direction from the start through the first study, and designed and built the toolkit we used to see activity patterns. Then I moved to building R&D partnerships with Nest on automation and intelligence.

What does this unlock?

01

Richer signals for routines

Identify and initiate routines based on spatial movement throughout the home. Predict upcoming interactions based on a spatial understanding of activities and routines.

02

Whole home automation

Coordinate automations across every occupied room, so activity in any room doesn’t interrupt people elsewhere.

What we produced

Zone Framework

We extended the idea of zones to give home intelligence systems more context.

Studies & Toolkit

We developed a kit called Cuboro to study how multiple Soli sensors work together across the home, alongside an analytics toolkit to visualize the data.

Use Cases

We applied the zone framework to a high-value use case: a multi-person household going to bed.

Zone Framework

HOUSEHOLD ZONES

A mental model of household zones

Zones are the places in a home that matter to everyday activities. After waking up, someone might move from the bed zone to the shower zone to the dressing zone. If several Soli devices understand these repeated paths, the home gains better context.

COMPLEX SPATIAL INTERACTIONS

Sequences

Many activities involve movement between a few different zones. Routines such as starting work or going to bed involve several zonal interactions before arriving at a final zone. These recurring spatial patterns are called sequences.

Rhythms

In real homes, several people often run sequences in different rooms at the same time. We call this whole-home view of concurrent sequences the rhythm of a home.

Studies & Toolkit

How could we use the presence ML algorithms of several Soli sensors to visualize zones, sequences, and rhythms during complex, home-wide activities?

STUDIES

Our Cuboro kit connected multiple soli devices to identify patterns

By utilizing new Soli ML algorithms, Soli for Spatial Awareness, experimented with Soli’s ability to track users in a room scale environment. During this project, we expanded our presence framework to explore how multiple Soli sensors could form a network of presence data for contextual awareness.

ZONES

Spatial density maps help identify zones

Our first experiments tracked a repeated coffee-making routine over a month. From the data, we built spatial density maps showing how long people spent in each zone of the tracking area. Warmer colors mark the areas that matter; mid tones show the paths people commonly use between hotspots.

We started with a coffee routine because it used at least two Soli sensors, had several steps, and was easy to repeat in different homes.

Repeated over several days, the same routine produced similar areas of interest.

SEQUENCES

Movement between zones over time help identify sequences

Zones on their own show areas of interest. To find activities over time, sequences work better: by tracking how long and in what order people move through zones, spatial sequence bars start to reveal repeated activities.

This bar shows the sequence of a coffee routine.

RHYTHMS

Several Soli devices help identify the rhythms of a home

With Multi-Soli, we could track several people’s sequences across the home at once, such as someone making coffee while someone else watches TV.

As people move through zones and start known activities, Multi-Soli recognizes them and accounts for everyone’s context, while preserving each person’s privacy.

Use Case: Sleep

We applied the framework and toolkit to the home’s most complex state: Sleep

NEST AT HOME

Multi-soli for whole-home sleep intelligence

With Soli in a growing number of Google Nest products, we could understand spatial context in more detail than before. Knowing each device’s type and location helps build richer context and predictions for Nest’s context engine.

Sleep rhythms extend beyond the bedroom

Entertainment zones

Public zones for entertainment and relaxation. Device: Television

Transient zones

Circulation zones that connects rooms to each other. Device: Thermostat

Rest zones

Pivotal zones for rest and sleep in the home Device: Hub

USE CASE 1

First person goes to bed

In multi-person households, people often go to bed at different times. Here we used spatial sequencing to detect User 1’s journey to sleep while User 2 stays awake in the living room.

USE CASE 2

Last person goes to bed

Here we used spatial sequencing to visualize what Soli sees as User 2 falls asleep next to User 1, who is already asleep.

By comparing User 2’s movements across the home with existing spatial data, Soli can tell when a second person has gone to bed, and that two people are now asleep.

These sequences are usually treated as rigid, personal journeys, but real homes are more fluid. With context about who is asleep where, the home can coordinate automations for everyone.

Experiences

What we unlock with zones

Security

First person to bed: All security notifications coming from the Nest Cam Outdoor device are silenced in the bedroom and sent to the living room only.

Last person to bed: Only emergency security notifications are now sent to the bedroom where both users are resting. All others are muted.

Lighting

First person to bed: Bedroom lights dim to ultra-low, the best conditions for sleep, while leaving enough light for anyone else who comes in.

Last person to bed: All lights across the home turn off. They respond to movement if anyone gets up during the night.

Entertainment

First person to bed: The TV volume drops from 80% to 60% to limit disturbance in the bedroom.

Last person to bed: Any devices still on are switched off, including the paused living room TV.

State of the home

What can happen when both people are asleep

Entertainment

When user 1 gets into the room, Google launches music or podcast based on listening history/patterns.

Assistant

Google assistant adjusts its voice to a whisper or low volume.

Climate and Energy

Thermostat and 3P climate devices devices i.e. air conditioners and fans, go into sleep temps to save energy and support healthy sleep until wake.

The opportunity here is that the zones and the information they provide can empower intelligence and help unlock key areas

What happened next: we released a design and development toolkit to visualize activity patterns, presented the findings at the Google Ambient Summit, and the work influenced Nest’s collaboration and investment in long-range sensors.

Contributions

Product design direction, strategy, research, interaction design, front-end development, project management, workshop facilitation

Collaborators

ATAP: Lauren Bedal (Interaction Design), Leonardo Giusti (Head of Design), Eiji Hayashi (TL HCI), Ryan Mather (Interaction Design), Timi Oyedeji (Interaction Design), Alex Paniutin (Software Engineer), Ivan Poupyrev (Director),

Google Home: Chrisoula Kapelonis (Design Strategy), Ron Morris (Design Lead)

Patent

System of Multiple Radar-Based Gesture Detectors, 2022