Organization and exploration of heterogeneous personal data collected in daily life

Many research topics, such as lifelogging, and personal information management, focus on the collection and the management of personal data. Extensive research on lifelogging has recently been carried out to collect vast amounts of personal data [1–3]. The personal data include email messages, schedules, Web sites visited, credit card payments, and photographs taken. They also include images, videos, sounds, and bio-sensor data. Most conventional research on lifelogging has been primarily concerned with the capture of personal data. It has also focused on building personal data archives [4].

Various personal data are stored in a variety of distributed sources, such as email messages, photographs on the WWW(World Wide Web), SMS(Short Message Service) on mobile phones, and perambulatory histories monitored by using GPS(Global Positioning System) embedded in mobile phones. There are also weight scales that connect to the Internet to store a user's weight on the WWW. It is expected to make wide use of smart meters that monitor the energy of homes by way of the WWW, such as the Google PowerMeter [5]. A variety of these personal data can be collected in the near future even if special devices that have cameras, microphones, and various sensors embedded are not always worn.

This paper focuses on reusing personal data for recall and helping users find various personal data and related information. This paper also describes methods of organizing and interacting with personal data. Personal data are heterogeneous. In other words, they contain a variety of media, formats, and granularities. Hence, it would be better to organize them by effective viewpoints in order to explore interactively rather than use the usual keyword searches. Moreover, various landmarks that trigger different personal data and related information are reported.

First, some viewpoints and views for organizing personal data are explained. Second, summaries and landmarks of data are introduced. Third, a visual user interface for exploration of personal data are proposed. Finally, a prototype system is explained, followed by a discussion on related work and our conclusions.

The concepts and user interface we propose were implemented in a prototype system. It was applied to nine types of personal data: photographs, GPS history, microblogging, schedules, web mail and SMS text messages, telephone call history on smart phones, numbers of steps walked per day as measured with pedometer, body weight measured every day, and home energy cost and use.

Parts of these personal data were collected from Web services, such as Flickr [15], Twitter [16], Gmail, and Google calendar. The other data were obtained from mobile devices and entered manually. We used iPhone and iPad as client mobile devices. iPhone was mainly used to collect personal data including GPS histories. iPad was mainly used to explore personal data with native-application user interfaces.

News topics (e.g., those from Yahoo! News) were used as one of the public landmarks.

This prototype was implemented mainly for demonstrating a feasibility of visualization and interaction with heterogeneous personal data. It basically used over 15,000 pieces of personal data from a test user and a lot of data from public including Flickr and Twitter. They were collected for more than a year.

Figure 5 has a system overview. Server-side modules were implemented using Java ^TM and the MySQL database. The 'personal data collection' module was used to collect personal data from various Web services. The role of the 'data mining' module was to execute data clustering and to calculate outliers that will be described later. The 'request handler' module was used to handle client requests and make responses by retrieving personal data. Client applications including 'data explorer' and 'GPS data collection' modules for iPhone and iPad were native applications implemented in Objective-C. This prototype provided several views for exploring personal data, such as map views, calendar views, and digest views.

The map view displays personal data according to their locations as shown in Figure 6. Unfortunately, only a few types of data could be automatically obtained from location information. Therefore, the location information (i.e., longitude and latitude) of personal data was approximately calculated by matching timestamps to GPS histories. Only representative locations were displayed on an initial display based on the result of clustering by latitude and longitude. The representative location was defined as the center of each cluster and it was one item of landmark information. GPS histories gradually appeared while zooming in on an area on a map, and personal data related to the area were displayed.

A calendar view provides a familiar view as is usually seen in a schedule book. Users can switch from yearly views, monthly views, and daily views. Figure 7 is a screenshot of a monthly view in a calendar view. An area corresponding to the day displays some personal data on the day. The right of the screen lists personal data on the selected day, when a user clicks one of these days.

Digest views were implemented in the temporal zooming user interface we propose. Figure 8 has a screenshot of a digest view. Photographs, visual charts, representative locations, and home energy costs are displayed at the top of the view, which is the main view. The photograph with the highlighted border is a landmark. Some text tags that characterize personal data during the period of time are displayed at the bottom of the view. Here, the tags are visualized as a tag cloud interface. These tags are also landmarks. When users click tags, related data on the main view are highlighted.

A digest view initially displays a summary of personal data on a given date and time scale as shown in Figure 9, which represents the hierarchy of a digest view in this prototype. The others appear while interacting with the digest view. For example, related photographs appear when the landmark photograph is clicked as was previously explained. Figure 10 shows a screenshot where the figure at left is an initial view that is a summary for May 2010. The figure at right shows another view for May 2010 after related data have appeared.

Public landmarks that are related to the period of time are displayed on the right of the main view. News topics during the period of time have been displayed in this example.

Figure 11 has screenshots of zooming operations. After users zoom in on a monthly view, a daily view appears. The day in the daily view corresponds to the date in the center data in the monthly view while zooming in. After they zoom out of a monthly view, a yearly view appears. The year in the yearly view corresponds to the year in the date in the monthly view. Moreover, the view changes into the display for the previous month by flicking the view to the right and to the next month by flicking it to the left. Of course, users can move to the daily view by clicking the data on a monthly view. The daily view corresponds to the date on clicked personal data.

One of the other landmarks is an outlier value for time-series data, such as the number of steps walked, home energy use, and body weight. Figure 12 is a screenshot of an outlier of the number of steps walked in a month. When users click the highlighted bar that indicates an outlier on the chart, the daily view for the corresponding date appears. Since the user in this example went on a picnic, the number of steps was more than those walked on other days. It is possible to create landmarks for values in data greater than a threshold to track records, such as those on body weight, blood pressure, and savings.

Figure 13 shows a variety of views for time-series data such as the number of steps walked. As previously described, a view is changed and determined depending on the time scale. In the figure, (1) indicates the number of steps walked per day specified by the text label, (2) indicates the number walked everyday per month specified by the bar chart, and (3) indicates the average number walked per day for a year specified by the text label. Future work is for an appropriate view to be automatically selected according to personal data and the time scale.

In Figure 14, when a user selects a photograph on a daily view, photographs that other people took on the same day and place are shown. Users seem to find new facts or reminisce about the past from other people's personal data. Here, only an example of photos being shared is described. Other shared data should create possibilities of people communicating with one another and facilitate the recall of fond memories.

Access control was not extensively studied in our current research. We need to safely manage permission for metadata and information on authorization. Since personal data are collected from diverse services, permission to use data is different from the original and complicated. Therefore, important work for the future is to study the management of permission and authorizations including research on OpenID [17] and OAuth [18].

Further study on summaries and landmarks is another important area for future work. Clustering algorithms for the content of data and attributes other than timestamps and locations should be studied. A search function is also necessary to enable explicit memories to be quickly and easily found. Taking into consideration the studies reported in this paper, we have to study several search functions, such as temporal, geographical, keyword, and spatio-temporal searches.

Social functions including synchronous and asynchronous communications are also important. Asynchronous communications using personal data acquired previously have vast potential in the future.

It is difficult to compare with existing approaches, because most of them are closed systems that we cannot try to evaluate them. Limited information could be obtained through single media, such as photographs. Data collected only from single device, such as PC, lack a variety of user experiences. In our research, not only photographs but also time series data, text messages, news topics, tags and combination of them facilitate a memory recall. Also, diverse perspectives are provided to users from a variety of media, different viewpoints (i.e., time, location, and people), and several landmarks.

Of course, detailed evaluations by users are very significant future works. Through developing the prototype and trials, the more the types of aggregated data increased and the more anxiety about information leaks and invasions of privacy increased. We will conduct a wide range of user tests with considering privacy issues deeply.

MyLifeBits is a system for storing lifetime data on a database [4]. It stores data from personal computers and photos taken by SenseCam, which is a mobile device that has a camera module, digital light sensor, temperature sensor, and passive infrared sensor [19].

Eagle et al. proposed a 'reality mining' system that measured information access and use within different contexts, recognized social patterns in daily user activities, and inferred relationships [20]. They used standard Bluetooth-enabled mobile phones. These researchers focused on collecting data with special devices.

Several user interfaces for memory aids have been proposed. iCLIPS provides a search user interface for logs from personal computers and photos taken by SenseCam [21]. Visual Augmented Memory (VAM) tried to show cues that were who (face), where (room), when (timestamp), and what (any visible action) through facial recognition from images stored on a mobile computer [22]. Autoalbum automatically generated a photo album by clustering photos based on the time they were created and the order in which they were taken [12].

Experience Explorer is a personal computer client that represents user data in a time-oriented manner [23]. All user activities (e.g., content generated, phone calls, tracks, music that was listened to) on mobile phones appear linearly arranged under user names. Lines of the user's friends appear next.

MemoryLens Browser employs inferences about landmarks in visualizations for browsing files and appointments [11]. Landmarks were predicted from a user's calendar data and computed as atypical organizers, atypical attendees, and atypical locations. To compute the value of locations atypical for events, they computed the number of times each location had appeared in a user's calendar over a fixed period of time.

The Stuff I've Seen (SIS) interface provides an integrated view of files on personal computers [24, 25]. The files were filtered with five fields of document titles, dates, ranks, authors, and mailtos.

Ringel et al. explained personal landmarks and public landmarks [26]. Personal landmarks were important calendar events and first photos taken on given days. Public landmarks were national holidays and important news events. Only files on personal computers were used and the time scale for landmarks was fixed in their research. In our research, several landmarks were suggested and appropriate landmarks were presented with changing time scales.

PERSONE is a Web-based life log media browser in which videos and audio are gathered by a special gadget. It can be browsed with a conventional timeline view and a map view [27].

Zheng et al. proposed a recommendation system that recommends activities using GPS histories and also recommends locations using user activities [28]. This seems useful in limited situations, because only small parts of personal data are used to analyze user activity patterns.

These researches tried to develop special devices, several visualizations and some activity analyses for a challenging and ambitious objective that is to collect and store all data of life and experiences. Since mobile communications and sensor networks such as IOT(Internet of Things) are becoming popular in our daily life, we have to study more natural and easier ways to collect and use personal logs.

Our approach expands previous researches and gives importance to help users recall and reminisce past events by integrating a variety of personal data in daily life with non-specialized devices and natural ways.

A study of the exploration of personal data was explained in this paper. A variety of viewpoints, views, and a temporal zooming user interface was described. Summaries and landmarks for memory cues were also presented. They were, e.g., representative photographs, outliers of time-series data, and locations. The methods we proposed enable users to recall and reminisce their memories and experiences.

Also, a prototype system in which our concepts were implemented was presented. It could be used to explore personal data including photographs, email messages, GPS histories, Tweets, histories of body weight, and home energy use. Further, a variety of personal data have to be integrated to study other views in the prototype. Detailed user evaluations and studies of other types of summaries and landmarks will be an important focus in future work.