ScreenTrack: Using a Visual History of a Computer Screen to Retrieve Documents and Web Pages

ScreenTrack: Using a Visual History of a Computer Screen to Retrieve Documents and Web Pages


Computers are used for various purposes, so frequent context switching is inevitable. In this setting, retrieving the documents, files, and web pages that have been used for a task can be a challenge. While modern applications provide a history of recent documents for users to resume work, this is not sufficient to retrieve all the digital resources relevant to a given primary document. The histories currently available do not take into account the complex dependencies among resources across applications. To address this problem, we tested the idea of using a visual history of a computer screen to retrieve digital resources within a few days of their use through the development of ScreenTrack. ScreenTrack is software that captures screenshots of a computer at regular intervals. It then generates a time-lapse video from the captured screenshots and lets users retrieve a recently opened document or web page from a screenshot after recognizing the resource by its appearance. A controlled user study found that participants were able to retrieve requested information more quickly with ScreenTrack than under the baseline condition with existing tools. A follow-up study showed that the participants used ScreenTrack to retrieve previously used resources and to recover the context for task resumption.

Productivity; Task resumption; Self-tracking.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
CHI ’20, April 25–30, 2020, Honolulu, HI, USA.
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-6708-0/20/04 …$15.00.




<ccs2012> <concept> <concept_id>10003120.10003121</concept_id> <concept_desc>Human-centered computing Human computer interaction (HCI)</concept_desc> <concept_significance>500</concept_significance> </concept>


[500]Human-centered computing Human computer interaction (HCI) \printccsdesc

1 Introduction

It is common for users to use multiple applications (e.g., programming tools, office software, web browsers, calendars, communication apps) simultaneously when carrying out computing tasks [2]. However, users often have to switch from one task to another, and they may be interrupted before completing a task [12, 20]. Users therefore need to frequently retrieve digital resources — e.g., software, files, and web pages — that they used in the past [16, 26]. This retrieval process can be time-consuming and cognitively challenging. A needed file may be located in a poorly organized folder structure, and the file name may not be informative (e.g., /Users/janedoe/Desktop/00291294.pdf). One common approach to this problem is to check the history of recently opened documents (or web pages) — file names or URLs sorted in chronological order, for example — in a particular application. However, there may be too many documents to choose from, and it may be difficult to recognize the correct item by its path, title, or URL alone. Furthermore, there may be other resources associated with a given primary document scattered across applications, and these are not connected to the document unless users manually organize such resources for each task they perform, which is challenging given the dynamic nature of modern knowledge work. Therefore, these additional resources can be easily lost in time or in multiple open windows. Eventually, users may need to rely on their memory, be forced to reopen and sort through a large number of resources, or fail to retrieve the resources entirely. The overhead in task resumption can be greatly reduced if users can retrieve primary documents and relevant resources efficiently.

Figure 1: ScreenTrack: a desktop application that captures screenshots at regular intervals to let users retrieve previously used software, documents, and web pages from the screenshot. ScreenTrack leverages users’ episodic memory and provides multimodal cues (visual and metadata) for users to recognize digital resources and retrieve them. For example, a user may not remember an author’s name, the URL of a page, or the title of a web page that they visited, but they may be able to recognize an image they saw. (A) A time-lapse video of computer screenshots; (B) UI controls for temporal navigation patterned after a video player; (C) metadata, such as application names, URLs, page titles, and filenames for the frontmost (active) application; (D) button to open retrieved items; (E) playback speed control; (F) date picker and close button

To address this problem, we present a way to retrieve previously used digital resources — software, websites, or files — from a visual history of a computer screen. We prototyped the method in ScreenTrack(Figure 1), a desktop application which records screenshots and lets users open digital resources present in the screenshots. We draw ideas from previous works that use a series of computer screenshots, or thumbnails of applications, to help users reconstruct their mental context [48, 51, 49, 50, 30]. ScreenTrack takes screenshots regularly, at a user-specified interval (e.g., every 10 seconds) to record current computer activities. It captures any metadata available from the operating system, such as the frontmost application, the path to an open file, or the URL to an open web page, and associates that information with the screenshot. ScreenTrack then provides users with a time-lapse video made of captured screenshots for a limited period of time (N days) from which they can navigate the visual history and reopen closed software, documents, or URLs (see Figure 1).

ScreenTrack is particularly designed for a short-term scenario; ideally, a user knows the approximate time when a document was open and can visually recognize it, but cannot remember the details needed to retrieve it (URL, file name, search keywords) for task resumption. It is worth noting that the current design choice of a video player-like interface may not be ideal for users to locate a screenshot from a lengthy history. In addition, the method is most effective when documents are visually distinct, as opposed to relying on textual content. These limitations will be discussed to motivate other novel designs for screenshot-based information retrieval methods.

To validate the effects of ScreenTrack, we conducted two in-lab studies. The first user study tested the effects of retrieving a web page using our system compared to conventional approaches; namely, examining the browser history or navigating the web to reopen a page. The results indicated that ScreenTrack decreased the time it took to retrieve a web page and increased productivity by 35.9% on average, compared to the control condition (without ScreenTrack). In the second study, we validated the software in the context of task resumption, where participants visited the experiment over two sessions with an interval of at least 48 hours between visits. The results showed that ScreenTrack helped participants quickly reopen primary documents, reconstruct their mental context, and retrieve relevant resources used during the first visit.

The research contributions of this work are as follows:

  • The idea of using a visual history of screenshots to retrieve digital resources, such as software, files, and web pages, that users want to reuse

  • Exploration of design goals for this method through the development of ScreenTrack

  • Quantitative and qualitative evidence that demonstrates the effectiveness of the system in retrieving digital resources using a visual history of a computer screen.

The results of this study reveal the design challenges involved in implementing such a method, and they help us understand users’ positive and negative perceptions (concerns about privacy and disk space) on self-tracking in computing tasks.

2 Envisioned Interaction

In this section, we present an overview of an envisioned interaction with ScreenTrack in a hypothetical scenario.

Sasha decided to move to a two-bedroom apartment so that she could have an extra bedroom that she can use as both a study room and a guest room occasionally. She was debating whether she should buy a sleeper sofa, which looks like a regular sofa. She typed “sleeper sofa” into a search engine and started surfing the web to find and buy one that she liked. She browsed tens of products and came across a daybed that she liked because of its contemporary look. However, she thought this daybed looked somewhat uncomfortable. She decided to keep looking at other kinds of sleeper sofa, eventually finding one that seemed reasonable, if less than perfect. She decided to think it over for a few days, since furniture is expensive.

Over the weekend, Sasha had a chance to visit a furniture store to see more options. She lay down on a few different daybeds and sleeper sofas and realized that a daybed is actually not that bad, given that she usually prefers a firm sleep surface. In the worst case, she could switch bedrooms when a guest spends the night at her apartment. The daybeds available at the furniture store were not in her price range, so she decided to go for the one online that she had liked. That night, she opened a web browser and tried to surf the web to revisit the web page that showed the daybed. However, she could recall neither the name of the daybed nor the website where she found it. Sasha went through her browsing history and tried searching the web for shopping sites with the keyword “daybed,” but the search returned a lengthy list of web pages. Instead of visiting each web page one by one, she opened ScreenTrack and opened the screenshot history from the first day of shopping. She knew that she had browsed furniture before going to bed, so she started working backwards through her screenshot history by scrubbing the timeline (Figure 1-B). She was able to recognize the exact product that she liked and realized she had never heard of the website before then, thinking, ‘No wonder I couldn’t remember where it was.’ She clicked the Open button (Figure 1-D), and the web page opened in her web browser. She added the product to her cart right away, without a second thought.

3 Related Works

ScreenTrack builds upon prior work on: (1) facilitating resumption of interrupted tasks, (2) organizing digital resources by task and time, and (3) using visual histories for mental reconstruction. We discuss prior work in these domains to provide context for the design choices made for ScreenTrack.

3.1 Strategies for Resuming Interrupted Tasks

Given the ubiquity of information and communication technologies, external interruptions are inevitable — email alarms, unexpected encounters in a work space, notifications that prompt users to check a smartphone or other device, and so on [12, 20, 27, 26]. There have been multiple approaches to address this problem: suppressing sources of interruptions [14, 45, 33, 42], controlling the timing of interruptions [1, 25], and facilitating resumption of interrupted tasks [16, 29, 43, 54, 56]. Among these, the idea implemented in ScreenTrack falls into the last category, since it aims to facilitate task resumption by reducing the time it takes.

Researchers have studied the various cues that knowledge workers use to resume interrupted tasks effectively. Their findings suggest the importance of visual cues in facilitating task resumption. Iqbal and Horvitz found that the amount of visual cues (viz., windows that are not obscured) corresponding to a suspended task serve as a reminder to resume the task [26]. Similarly, application windows that are left open can be a cue for programmers to recover the context of a task [31]. Jo et al. developed gaze-based bookmarking, significantly reducing the time taken to resume reading [29].

Task resumption has been studied actively in the context of software engineering, as programming is a complex activity that involves multitudes of files and resources. Parnin and Rugaber found that programmers rely on the most recent visual state for resuming interrupted programming tasks; programmers tend to run their programs and navigate code to resume tasks [43]. Researchers investigated the effects of displaying programming activities in chronological order, in terms of both success rate and subjective ratings from users [44]. In particular, the authors evaluated participants’ perceptions of the potential of providing an instant replay of screenshots, which is the core idea of ScreenTrack. This approach of replaying a code change history has been an active topic in software engineering, as an alternative to coarse-grained version control systems [23, 59]. Existing literature suggests that being able to replay changes in a (reverse) chronological order can be a rich cues for resumption of complex tasks.

3.2 Organizing Digital Resources by Task and Time

Organizing documents and files has been a challenge for knowledge workers. Researchers have developed methods to organize digital resources by task or by activity. For example, GroupBar lets users manually organize opened documents by task and makes it possible to restore or minimize batches of documents, which can be useful to quickly switch between tasks [56]. Taskposé visualizes multiple application windows whose proximities are calculated based on usage patterns and transitions between them [5]. TaskTracer implements a similar approach in an automated fashion whereby the program keeps track of user activity and system events to associate documents with a particular task [16]. While this can be an effective solution to organize resources, users may need to give additional consideration to the concept of a task, and what constitutes a task may vary across contexts.

In contrast, a number of applications have taken temporal approaches to archiving and retrieving digital resources, diverging from the task-centric approach. LifeStreams takes an approach that suggests documenting electronic files temporally, a concept that is prevalent in modern software with familiar features like “recent documents” or a browsing history [19]. The idea of time-machine computing is to allow users to go back in time and retrieve files from a previous version of the desktop, similar to the backup system of the same name included in macOS [47]. Laevo is a temporal desktop interface that can organize active applications and web pages based on activities laid out temporally, in a manner resembling a Gantt chart [28]. For example, by clicking a box that represents an activity from the past or which is planned in the future, one can retrieve all of the activity’s associated applications and digital resources. YouPivot is yet another system that leverages the temporal proximity of a task to another task and enables contextual searches[22]. ScreenTrack is similar to these time-centric tools insofar as it leverages humans’ episodic memory with rich contextual cues from adjacent tasks and lets users locate a time window of interest. However, it also provides rich visual cues to help users reconstruct their mental context and recognize digital resources, whereas the other tools reviewed in this section rely on symbolic information(text, icons).

3.3 Using Visual Histories for Mental Reconstruction

Many previous studies found potential in using visual histories of activities to facilitate task resumption by helping users reconstruct their working contexts. For example, participants could determine whether they saw a given picture from an extensive collection over a two-day interval with 90% accuracy [57]. In other studies, human could spot even a subtle change [6] and recall detailed context from an image [55]. Researchers have made effective use of the capability to recall and recognize visual images in a productive setup. One approach to reconstructing a past working context is to provide users with memory cues in the form of visual information in chronological order, such as by writing “biographies” with icons [35], recording video footage of an office environment [13, 34], archiving a user’s web browsing history [50], or identifying task boundaries in programming environments [52]. Some systems provide multimodal replay (viz., visual and symbolic), which can not only help users recover context visually, but also retrieve a symbolic state for learning or collaboration [21, 37, 59, 18]. In particular, retrospective cued recall, which uses screen recordings of computing tasks, has been employed as a usability testing method and found to be accurate even with lengthy delays [7, 51, 50]. Recently, Rule et al. reconfirmed the benefits of using a visual history for mental reconstruction and enumerated the design implications of such software by exploring various design options (animation vs. still image, screenshot size, crop region) and their effects [49]. Beginning from the idea of leveraging the benefits of using a visual history—already confirmed by numerous researchers—we extend it by using a visual history as a method of retrieving digital resources for end users.

4 SCREENTRACK: Design goals

To test the idea of using a visual history of a computer screen to retrieve digital resources, we designed and implemented ScreenTrack. In this section, we discuss the design goals for effective retrieval of digital resources.

4.1 Presenting Multimodal Cues in Temporal Order

The core idea of ScreenTrack is to provide a time-lapse video of screenshots that permits users to later open the frontmost (or active) application shown in the screenshot. ScreenTrack leverages humans’ capability to reconstruct a mental context and recall contextual information from a visual history [49, 50, 51]. Existing methods of recording history stored in different software operate using primarily textual information. Some categories of information (e.g., file names, folder structure) must be manually created to be informative, while others are out of users’ control (web page titles, URLs). As users may find it challenging to recall the actual contents of a document from such abstract information, ScreenTrack complements existing methods of retrieving digital resources by giving users visual cues in temporal order. This means that ScreenTrack provides multimodal — both visual and textual — information. It combines screenshots with the textual information — or metadata — available from existing methods, including file and folder names, URLs, page titles, and application names. A series of studies has shown that displaying textual information alongside images can significantly improve both recall of past working contexts and comprehension of the details of previous tasks [15, 36].

Similarly, the time-lapse video generated by ScreenTrack distinguishes itself from pure screen recordings in two aspects. First, screen recordings only provide visual cues (implicit), whereas the multimodal cues available in ScreenTrack can offer users metadata (explicit). Second, a user can open a document from a screenshot in ScreenTrack, while existing recording software offers no means of recovering resources captured in a video at any level. i.e. one cannot copy text from a screen recording, whereas, in ScreenTrack, one can open a document from the video and copy text.

Lastly, we made significant use of the findings from Rule’s et al. study: we used full-screen thumbnails instead of cropped images, implemented timeline scrubbing, preferred still images over animations, and automated screenshot captures at a fixed interval [49]. Given the temporal representation of cues, ScreenTrack is patterned after a video player. ScreenTrack provides users with three different ways of navigating the visual history of their screens (See Figure 1-B): users can play the history as a time-lapse video with speed control (Figure 1-E); scrub through the timeline by clicking and dragging the slider or clicking a spot on the timeline (useful for locating a region of interest); and use buttons labeled < and > to display the previous or following captured image. Overall, a visual history of screenshots with metadata provides users with rich multimodal cues for them to easily reconstruct mental contexts, and to efficiently retrieve digital resources from the past.

4.2 Cross-application Retrieval from a Screenshot

ScreenTrack utilizes a visual history of computer screenshots and can inherently document any kind of software that appear on a screen. Most modern software provides a way to access a history of previously used files or web pages. For example, Google Chrome can record the history of web pages accessed by its user over the last (e.g.) 90 days, and Microsoft Word can show a list of recently opened files and documents. However, each application tracks recent items independently of others, so no application keeps tracks of cross-application usage of a computer and can provide cues for users to understand and utilize the implicit dependencies between files and resources: Google Scholar may always be open while a user is writing an academic paper, a stock photo site may always be open while a user is editing presentation slides, and so on. Rule et al. found that users made significant use of “implicit cues in their working artifacts to reconstruct mental context, particularly for relationships between artifacts” [49]. As one may need multiple documents and multiple web pages open for a task, the ability to find and recall resources associated with a task is essential [22]. As ScreenTrack present screenshots in chronological order and users interleave multiple applications within a task, they can retrieve resources that are relevant to their primary documents by navigating the timeline.

While a captured image of a computer screen inherently works for any kind of application that appears on a computer, capturing metadata is not trivial. Each application has different kinds of metadata meaningful to users, and it may not be readily available to other software. For example, in the case of a web browser, the URL and title of a web page are relevant to users, while in the case of a word processor, a document’s name and location on disk are relevant to users. Certain applications offer little or no metadata (e.g., Dictionary, Chess), and certain applications have the property of working with groups of files as one unit, as is the case when opening a project in an integrated development environment, or when opening a library of entries in a bibliography software. To make ScreenTrack work across applications, we categorized frequently used software (office/productivity suites, programming environments, web browsers) and designed scripts to capture metadata differently for each. In addition, labels shown in the ScreenTrack user interface are adapted to the category of the frontmost application for the current screenshot. For instance, if the active application is a web browser, the item label reads “Page URL” (as seen in Figure 1-C). If the active application is Microsoft Word, the item label will instead read “File Directory”, and the button next to the label may be activated to open the enclosing folder. The current version of ScreenTrack provides four different categories and covers most frequently used software. The list of categories and applications is detached from the main algorithm so that it can be easily updated from a central repository of application-category mapping. In summary, metadata can be captured across applications in ScreenTrack.

4.3 Addressing Users’ Privacy Concerns

Even though more people are comfortable with recording their personal data in the emerging culture of self-tracking and the quantified self [8, 17, 41, 32], there remain critical concerns that self-tracking technologies may put one’s privacy in danger [4, 38, 46]. We designed ScreenTrack to address the potential concerns involved in recording a computer screen, given that personal information (such as a credit card number) may be displayed on-screen, as anticipated in the previous study [49]. Firstly, ScreenTrack is designed to use only local storage (instead of cloud-based storage). Therefore, cross-device functionality was not considered (such as stitching videos from multiple computers), since these features would require captured images to be transmitted from one device to another. ScreenTrack is meant for use on a single computer. In addition, all the images captured in ScreenTrack are designed to be accessed only through the software, not through external means. Only those explicitly granted access to the computer can review the time-lapse video, assuming the computer is password-protected. In addition, it is trivial to incorporate them the software in ways that conform to modern security standards — giving users the option to protect the software with a password or biometric authentication and using encryption to deny third parties access to captured images. Currently, we plan to implement these features for a future long-term deployment study and the eventual release of the software. Lastly, the time-lapse video is designed to be ephemeral. Storing only recent data (recent days) may alleviate some users’ concerns about privacy [3, 58].

Figure 2: ScreenTrack configuration: Users can configure the recording parameters and control how much disk space is needed. Clicking Open Time-lapse Video will open the main interface shown in Figure 1.

4.4 Moderate Disk Space Usage

ScreenTrack is designed to use a minimal amount of disk space, given the effectiveness of a visual history in mental context reconstruction and retrieval of digital resources. In theory, as always-on screen recording software, captured screenshots could easily occupy all available disk space within a few months. Importantly, making the time-lapse video ephemeral and limiting the period for which ScreenTrack will keep a time-lapse video are essential in limiting the amount of disk space used. This approach is common with other kinds of recording software and equipment (e.g., dashcams, security cameras, action cams). Users can also configure the time-lapse video quality (image compression and capture interval) to decrease the amount of disk space that the screenshot repository will require. However, considering the trade-off between the quality of the captured visual history and the amount of space needed, ScreenTrack imposes minimal values for some parameters (such as a minimum dimension of 320 px and a maximum capture interval of 60 seconds) based on the findings from a previous study [49].

In addition, ScreenTrack implements a few algorithms that can reduce the total amount of disk space needed. For example, Figure 2 shows how one can choose to capture a screenshot whenever the active application is switched, so that one can choose to specify a longer interval without missing the opportunity to capture each application used, similar to the motion-activated security camera. In addition, ScreenTrack detect if the newly captured image is worth storing. If two consecutive images are identical, it stores only one image, given that the following image does not add any new information. The single image is instead reused for several frames of the generated time-lapse. Using the capture configuration, ScreenTrack provides a conservative estimate of the amount of disk space required, given a users’ input on the length of video storage(See Figure 2). For instance, based on current practice, approximately 20 GB are needed to store 80 hours’ ( 10 days’) worth of data, given a reasonable configuration: screenshots every 10 seconds, 80% compression (assuming a 1440 1080 capture resolution). Informing users of approximate disk space requirements enables them to reconfigure the software to strike a balance between capture length and resolution (i.e., both time and size) of the replay, and to thus control the disk storage space they would like to dedicate for ScreenTrack.

4.5 Implementation

We designed and implemented ScreenTrack in Swift for MacOS devices. The program also uses AppleScript and the Accessibility API to capture metadata, launch software, and open documents and websites.

ScreenTrack saves two kinds of data in the recording process: captured screenshots and corresponding metadata. Each screenshot is stored as a JPG image, and all metadata is stored in a JSON file with time stamps and paths to the corresponding images. During use, ScreenTrack remains present in the menu bar as a camera icon. Clicking the icon displays a configuration window as seen in Figure 2. Users can specify the screenshot interval and per-image compression ratio before starting recording in the interface. Collected data is stitched or segmented by day. Users can access a time-lapse video by selecting a date (See Figure 1-F).

5 User Study Result

We conducted two in-lab studies to validate the effects of ScreenTrack in two different settings. Each studies investigate the following research questions (RQ1) Does using ScreenTrack reduce the time it takes to retrieve a digital resource from the past compared to existing methods? (RQ2) How does ScreenTrack support users in the context of resuming suspended task? (RQ3) What are users’ perceived values and potential concerns of ScreenTrack? Each question will be explored in each section.

5.1 STUDY 1: Revisiting a Web Page of Interest

Method: Recruitment and Study Procedure

We were interested to see if using ScreenTrack would cause a reduction in the retrieval time. This could lead to a productive gain overall in computing tasks by complementing existing methods, when there is no single, direct way (e.g., a bookmarked web page or a file on the desktop) to retrieve a necessary resource. Modern knowledge workers often encounter this situation. Study 1 is designed to investigate how ScreenTrack helps users retrieve a previously opened digital resource (a web page, in this case) from a screenshot. To that end, we simply measured the retrieval time, which is the time it takes users to retrieve a web page that they recall.

We recruited 20 university students (7 of whom were female) through a student mailing list at the authors’ university. The average age of the subjects was 26.2, ranging from 22 to 35. The results of the demographic survey indicated that all the subjects self-assessed their digital literacy as ‘Average’, ‘Above Average’, or ‘Excellent’, and they all frequently surf the web (at least once a day). We invited the subjects to a lab and provided them a laptop on which ScreenTrack was installed. Upon arrival, all subjects signed a consent form and filled out a demographic survey (See Figure 3).

In Study 1, subjects performed an online shopping task. We asked them to browse furniture products (such as couches, media stands, and coffee tables) for two different room types (a living room and a bedroom) until they had decided which products to purchase per type. We asked subjects to select at least three products per room within 25 minutes. The task was divided into two phases: (1) the web-surfing phase, in which subjects were asked to browse online shopping websites; and (2) the retrieval phase, in which a study moderator asked each subject for their favorite products per type and instructed them to retrieve the web page on which the chosen product is listed. Each subject did both tasks under two different conditions: the control condition and the ScreenTrack condition.

Figure 3: ScreenTrack The user study procedure. The user study was composed of seven parts: (1) consent form, (2) survey, (3) web-surfing task 1, (4) web-surfing task 2, (5) recall task 1, (6) recall task 2, and (7) exit interview.

(Phase 1) In Phase 1, we asked subjects to pick three furniture products that they would like to purchase. Subjects were told to assume that they were moving into an empty version of the home that they were currently living in. We split rooms across conditions for each subject; in one condition, we asked them to browse furniture for their living room; in the other, we asked them to browse furniture for their bedroom. The purpose of this split was to avoid having subjects pick the same set of furniture under two different conditions; additionally, there was a risk that subjects would run out of ideas if they were asked to choose furniture for the same room twice. Four combinations of room and condition were thus created: C1 (control-Living room), C2 (ScreenTrack-Bedroom), C3 (control-Bedroom), and C4 (ScreenTrack-Living room). Each subject was assigned one of two pairs, either (C1, C2) or (C3, C4). We counterbalanced the number of subjects per sequence of pairs, for a total of 5 subjects in each of four different sequences: (C1-C2),(C2-C1),(C3-C4),(C4-C3).

(Phase 2) Before Phase 2, everyone was introduced to ScreenTrack and went through a short, interactive tutorial on how to use ScreenTrack. We made sure that each subject knew how to open a document and a web page from a screenshot using the Open button and had tested each way of navigating through time (playing the time-lapse, playing it at a different speed, scrubbing through the timeline, using the buttons to show the next or previous screenshot). After Phase 1, all subjects were asked to answer the following questions per product category (for a total of three products): (1) Could you describe the details of your favorite product (e.g., couch)? (2) Could you open the web page that contains the product you just described? The first question was chosen to let subjects pick a single furniture product among the products they browsed. The second question was chosen to measure the time it took to retrieve the web page. For the second question, if they used ScreenTrack in Phase 1 for the product in question, we asked them to find the web page using the time-lapse video in ScreenTrack. In Study 1, we configured ScreenTrack with a screenshot interval of 5 seconds and a screenshot resolution equal to 40% of the screen’s resolution. For products found when ScreenTrack was not in use, subjects were free to use any conventional method, such as typing a URL, exploring the browser history, searching for keywords on any web page, and/or navigating to the web page by clicking hyperlinks and menus. In both conditions, we measured the time between the very first interaction with the computer — typically, launching a web browser in the control condition or launching ScreenTrack in the experimental condition — and the time of the user’s interaction that opened the target web page. We repeated the same questions for the second task completed in Phase 1. With 20 subjects, we collected 60 data points per condition (3 products per task, 20 subjects), making for a total of 120 data points with two different conditions. We conducted an exit interview to ask subjects general questions about the system, which we will present the result in the later section. We recorded audio and the screen of the laptop during the entire study session, from the point when subjects finished the survey until the end of the exit interview. We manually annotated how each subject retrieved the target web page, as well as how long it took. We double-checked that the page they retrieved actually contained the product described and confirmed that they visited the page during Phase 1 from the screen recording.

STUDY 1 Result: Time Reduction in Resource Retrieval

The result indicated that subjects can retrieve a target web page quicker with ScreenTrack than without it. Under the ScreenTrack condition, the average time to retrieve a web page of interest is 27.1 seconds () compared to 42.9 seconds () under the control condition. This is equivalent to a 35.9% gain in productivity.

We ran a repeated-measures ANOVA to test the significance of the effects of the following variables on retrieval times: (control or ScreenTrack), (the first, second, or third product chosen in each task, to test for a within-condition learning effect), (Bedroom or Living room), and (the first or second task, to test for a learning effect between tasks). Because distribution of the retrieval times does not satisfy the normality based on Shapiro-Wilk test (), we performed log-transformation of all data points and confirmed the normality of the result (Shapiro-Wilk test, ). The test result showed that the difference between the mean retrieval times for each value of () was statistically significant. This result indicates that using ScreenTrack reduced the time taken to retrieve a previously visited web site. In addition, mean retrieval times according to other groupings were also statistically significant: (), and (), indicating that subjects retrieved living room products more quickly on average. Meanwhile, () indicated that subjects completed the second task more quickly on average. We figured out that there is no significant interaction effect among the factors of our interest (all of the p-values > 0.05).

We further analyzed the data to understand the within-condition learning effect (Trial). This is to investigate whether consistent use of ScreenTrack could lead to greater benefits to productivity. Figure 4-A shows the average retrieval time per trial and confirms that subjects could more quickly retrieve a web page in later trials, especially under the ScreenTrack condition. We ran Tukey HSD test to pairwise compare the retrieval times in order to see the significance of the difference between the times across the trials. This revealed a statistically significant difference between the first trial (S1) and the third trial (S3) under the ScreenTrack condition (). A marginally significant difference between (C1-C3) and (S2-S3) was also found (). Based on the trend shown in Figure 4-A and the result of Tukey HSD test, we can conclude that constantly using ScreenTrack has capability to improve users’ performance. We reviewed the video recordings and found two potential reasons that may explain this trend beyond the learning effect.

Figure 4: The effects of Trial: (A) Average retrieval time decreased over trials, especially in ScreenTrack. (B) As the trials progressed, more subjects used the scrubbing method to browse the captured screenshots, which may account for the decrease in retrieval time in (A).

Firstly, we observed those who simply replayed the time-lapse video from the beginning took significantly longer to retrieve their page of interest than those who navigated the video using either the slider or the frame-by-frame < and > buttons. Some subjects scrub through the timeline until they find their product of interest, and may then press the < and > buttons to finely adjust the timing. This approach took less time than the approach in which subjects watched the replay from the beginning. To see the usage patterns over trials, we annotated and calculated percentages of control usage (replay, slider, buttons) across subjects. In general, note that the majority of people used the slider rather than replaying the time-lapse video. We can see that on average, the subjects used the replay function less and used the slider (timeline scrubbing) more in later trials, as seen in Figure 4-B. While it could not be confirmed statistically, increased slider usage as subjects became more familiar with ScreenTrack might explain why subjects became quicker at retrieving a web page.

Secondly, the way that ScreenTrack works gives users additional information as they use it. For example, when a user opens ScreenTrack, the slider will be positioned at the most recent screenshot relative to the selected date. One common pattern for using the horizontal slider (timeline) is to grab the handle and drag (or scrub) it to the left. As they drag the handle, the video view is updated to respond to the handle motion. Therefore, subjects were shown other products that they browsed before while they were trying to locate the product. Even though they did not know exactly where their products of choice were located on the timeline for Trials 2 and 3, subjects should have known approximately where other types of products (e.g., coffee tables) were on the timeline. In that sense, we strongly believe that this could be another reason for the continued improvement in retrieval times. The result of Study 1 demonstrated the potential benefits of using a visual history of a computer screen for the purpose of retrieving web pages.

Interview Result: Perceived Values and Concerns

In this section, we briefly summarize the result of thematic analysis we have done on the exit interview of Study 1. We asked four general questions (benefits, concerns, willingness to use in practice, open-ended suggestions) to learn their perception on the system and potential benefits and concerns that they found.

In the interview, 15 subjects (out of 20) said that the user interface of ScreenTrack is easy to use and intuitive. Subjects were favorable to the interface because it is similar to watching a video. A number of people commented about the benefits of multimodal cues and the feature that they can retrieve digital resources. Pr’s comment nicely summarize the benefits:


P4: "It is easier to use compared to the web browser history where I have to actively search for the keyword. It follows a chronological order that can even further help me back to chase what I did and which web page I clicked. of course, it can direct me right to the web page which is pretty cool and easy."

Half of the subjects (10/20) discovered the benefits of retrieving digital resources across application from a time-lapse video. They also agreed that the visual history can help users recall more details about previous working context. The following comments well resonates with the challenges of modern knowledge workers and describes how ScreenTrack address such a challenge and potentially enhance productivity.


P5: "Because I will write many papers, there are many files you know. Sometimes I forgot to rename them. So it is really difficult for me to find the exact files in a short time. (…) So with this tool, I think I can save lots of time to select, to find the files I want to even across many different applications."

Subjects raised a few concerns on using the ScreenTrack during the interview. Subjects worried that this software would record personal information and might leak this data to someone else, like credit card information or home address.


P16: "I mean it is recording me, so that’s a concern,(….) if it goes into somewhere then it might be in trouble, that’s a problem."

Lastly, in addition to the common concerns on the disk-space usage, quite a few number of subjects also expressed their hesitance in installing a new software to their computers. While these concerns were anticipated and considered from the design stage, we realize that there may be actual barriers of using ScreenTrack in practice. Features that can enhance awareness and accountability would be necessary to alleviate their concerns on privacy and computational resources.

5.2 STUDY 2: Facilitating Task Resumption

Study 2 is designed to investigate the potential benefits of ScreenTrack in more complex setups and over longer periods of time. The context of Study 2 simulates a scenario in which a user needs to resume a suspended task, like the example scenario given in the Envisioned Interaction. In summary, we asked subjects to create a deck of presentation slides over two sessions, with a minimum 48-hour interval between sessions. With this task, we were interested to see how subjects would retrieve the primary document and associated digital resources when resuming the suspended task.

Figure 5: ScreenTrack The procedure of STUDY 2.The user study 2 is composed of 7 parts: (1) consent form, (2) making slides task 1, (3) making slides task 2, (4) time interval, (5) resuming task 1, (6) resuming task 2, (7) exit interview

Method: Recruitment and Study Procedure

For Study 2, we recruited 8 subjects from the author’s university. The average age of the subjects was 25.1, ranging from 23 to 27. We asked subjects if they were familiar with presentation slide authoring software (e.g., Microsoft PowerPoint), and all subjects assessed their skill in such software as either “intermediate” or “advanced”.

Subjects in this study were asked to create two decks of presentation slides under two different conditions: control and ScreenTrack. One slide deck was to introduce Hawaii as a place for a family of four to travel, while the other was to introduce Los Angeles as a destination for subjects and their friends to visit. These topics were selected for their general relatability through a pilot study. The same laptop used in Study 1 was used for this study.

Study 2 was conducted over two sessions, simulating a situation in which a user has been interrupted and must resume the suspended task after a fixed interval of at least 48 hours. In their first visit, subjects were asked to perform one task for 15 minutes, and then stop the task. After stopping the first task, they began the second task, which was also interrupted after 15 minutes regardless of what they were doing. For each subject, one of the two tasks was recorded by ScreenTrack, which was notified to the subjects. In this study, there were two factors: Condition (control or ScreenTrack) and Place (Hawaii or Los Angeles). Each subject performed one of two pairs: either (control-Hawaii, ScreenTrack-Los Angeles), or (control-Los Angeles, ScreenTrack-Hawaii), in a randomly assigned order. The interruption was patterned after a typical external interruption, in which users are stopped suddenly and do not have time to wrap up their work properly [11, 39]. In the beginning of the first visit, they were asked to save the slide deck in a designated folder that the study moderator provided. While subjects were not familiar with the folder structure, it was nonetheless well organized and named, such that subjects could navigate it and save their files easily. For example, a slide deck for Los Angeles was saved to this folder: . The folder structure also contained a variety of other folders that served no real purpose, to improve the realism of the scenario. Lastly, we asked subjects to log into the web browser (Chrome) with their own account (i.e., a Google account) before beginning the task. This way, any browsing history made on the laptop would be logged to the subject’s personal account and later synchronized during the second visit. This serves to create an environment similar to that of using one’s personal computer; otherwise, all subjects’ browsing history would appear in the browser.

Each subject’s second visit was scheduled 48 hours after the first visit with the exception of one subject, whose second visit was scheduled 70 hours later than the first visit due to a scheduling conflict. The 48-hour interval was selected because it is known that within two days, one will forget roughly 70% of what one has learned without any reinforcement [40]. Upon arrival at the second visit, subjects were asked to resume the suspended task — completing the slide decks they created, in the same order as they began them during the first visit. For their control condition task, users did not have ScreenTrack, but were allowed to use any other functionality (e.g., file search, recent documents, file manager [the Finder on MacOS], browser history). For their ScreenTrack condition task, subjects were encouraged to use use any tools they desired, including ScreenTrack. The same tutorial (from Study 1) was repeated before running the ScreenTrack condition task. Each task was stopped within 15 minutes of its start time. Upon completion, we conducted the exit interview as in Study 1.We recorded audio and the screen of the laptop during the entire study session, from the point when subjects finished the survey until the end of the study. Later we manually annotated the screen recording video for analysis While we were also able to confirm the time reduction benefit in Study 1, we focused on what kinds of natural behaviors would emerge as subjects resumed the suspended tasks with ScreenTrack.

Figure 6: Resumption lag per subject. The label indicates (Condition, Subject ID, The method of file open). The left bars (dark) present the time it took for a subject to open the file. The right bars (light) present the time it took to make the first edit after opening the file.

Result: ScreenTrack Usage in Longer Time Scale.

The result suggests that using ScreenTrack is useful in reducing resumption lag overall. Resumption lag is defined as the time needed to resume the primary task, and is often determined by the first edit that a user makes to the primary document [53]. Figure 6 shows resumption lag per subject. We broke down resumption lag to additionally measure when and how the file was opened (the left side of the bar — dark dotted bar). For each bar, each label is composed of condition, subject ID, and the method that they use to open the file.Note that not all subjects (viz., P1, P2, P7) under the ScreenTrack condition used ScreenTrack to find their presentations file and open them from a screenshot; some of them, even after spotting the file in a screenshot by scrubbing through the timeline, habitually either clicked the application icon or opened the file manager to locate the file. We can observe that opening a file from ScreenTrack did help subjects quickly retrieve the file from a snapshot. Those who used “recent documents”-type functionality were similarly able to quickly open the file.

Once they opened their files, it took a while for subjects to make their first edit of the session. This is the time dedicated to mentally reconstructing the task context. As shown in Figure 6, overall resumption lag in case they used ScreenTrack is shorter than the ones in control condition with one exception (P8). We checked the screen recordings for all subjects in ScreenTrack condition, and all except one subject used ScreenTrack to review what they did, typically after reviewing their slides. While reviewing the slides did provide subjects with the most recent version of the artifact, it did not inform them of the nature of their most recent change was or what they were about to do. Watching the temporal history of screenshots might have helped reconstruct their mental context, which can potentially be a reason for the productive gain.

ScreenTrack provided richer information than the file opened. The visual histories always included the web pages that they used at the first visit. For example, a subject was interrupted when they were entering “family friendly restaurants in Honolulu”. The search keyword that the subject was able to see would have provided richer contextual information (e.g., a trip for family of four) about the task than the empty slide that reads “Top places to eat”. This kind of additional cues cannot be obtained from the primary document itself and would be the benefit only comes from cross-application cues available in ScreenTrack.

Figure 7: The usage of ScreenTrack per subject (sorted by the elapsed time) (Left Bar) indicates the number of times that subject used ScreenTrack, (Right Bar) indicates the number web pages open from ScreenTrack, (Line) The time(seconds) that subjects spent using ScreenTrack during the 15 minute-long task at the second visit

We also investigated how the usage of ScreenTrack vary across the subjects. From annotating screen recordings, we counted 1) the number of times that they used ( = 4.1, = 3.1), ScreenTrack, 2) the number of web pages opened from ScreenTrack ( = 1.6 , = 1.92), and 3) the time that they spent in ScreenTrack ( = 83.6 sec, = 49.9). Each subject’s results are displayed in Figure 7. The usage of ScreenTrack varied much across the subjects; i.e. P6 did not open any web page and almost did not use the application (5 seconds for initial file open), while P4 actively used the application to open six web pages that they used from the 1st visit, not just initially but throughout the session. On the other hand, P8 spent the most time (146 seconds) watching the visual history from ScreenTrack — 1/6 of the total time spent at the second visit. At the first visit, the subject spent most of the times searching information, rather than making slides. By the time that the subject was asked to stop the task, the slides was mostly empty and the works she has done during 15 minutes did not translate to the artifact. Therefore, the subject started with sparse slides and decided to watch the replay from the beginning to remind themselves what they did last time. The observation that some subjects spending time on watching replay throughout the session, and retrieving digital resources from the past shows the limitation of resumption lag as a representative metric to evaluate the effectiveness of task resumption. It is because resumption lag only highlights only the initial part of the challenge [53]. The variance of usage highlights the need of long-term deployment studies as a future work in order to understand how ScreenTrack can be integrated in their day-to-day computer works.

6 Limitation and Future Works

One limitation of ScreenTrack is that it does not scale well over the temporal dimension, for medium- or long-term retrieval. The current design of ScreenTrack — a media player-like interface — makes it difficult for users to retrieve a document from a lengthy history. For example, if a user cannot recall the exact date of an incident, it would be very time-consuming to go through multiple time-lapse videos to identify a snapshot. Even within a single video, locating a snapshot without an idea of its position on the timeline is as challenging as navigation of video streams to find information just by visual inspection. This timeline design limits the effectiveness of ScreenTrack to scenarios that involve resuming recent tasks, or cases in which users know approximately when a document was used within the last N days (e.g., finding a product in the evening after viewing it in the morning, or locating a website visited last weekend). Instead, ScreenTrack relies on a person’s episodic memory, which is the ability to remember events based on visual images and their temporal order [10]. Existing methods do not leverage a user’s episodic memory to retrieve resources, as they do not use images sorted in chronological order. In that regard, we believe using ScreenTrack has value in providing a new retrieval method different from and complementary to existing ones. However, it is worth noting that other types of design choices (e.g., query-based [24], sampled or curated snapshots) may be more effective for longer-term scenarios.This limitation motivates our future works, in which we plan to explore other design alternatives for screenshot-based task organization. In particular, we plan to develop a new visual representation of screenshots geared towards long-term usage of screenshot-based methods for document retrieval, mental reconstruction, self-reflection, and behavioral change in the context of productivity. One design choice we plan to explore is a collage of screenshots that an intelligent system can generate from ScreenTrack data. Such visualizations may be able to serve as representative summaries of what a user did in the past, and may also project how productive the user was [9].

Another barrier to extending the benefits of ScreenTrack is the limited ecological validity of the user study. While we believe evaluating the system in a controlled setting was inevitable to provide quantitative evidence, given the limited time spent in the lab, we may discover more benefits and challenges associated with ScreenTrack if tested in practice. For example, in real-world use, users would need to go over much longer videos, full of irrelevant information as described previously, which may slow down the retrieval process. We plan to run a long-term deployment study ScreenTrack and are in the process of developing practical features to address participants’ concerns (disk space, multiple monitors, encryption, password protection, tolerance to file path changes). Through the extended study, we plan to further explore and understand how users’ perceptions change in the long term. Specifically, we wish to study how ScreenTrack causes behavioral changes, to discover unexpected (both positive and negative) implications of self-tracking in the context of productivity. For example, if ScreenTrack can be seamlessly integrated into users’ computing activities, allowing them to retrieve documents, web sites, and applications, this may change the perceived value a cluttered workspace has to a user as one motivation to leave documents open is not to lose them. Users who currently leave many applications running at once and open dozens of browser tabs may feel more comfortable closing unused resources.

Lastly, we note that the suggested method heavily relies on the visual nature of applications and websites. The current user study only involves computing tasks that are designed around visual tasks (online shopping, authoring presentation slides) rather than text-heavy tasks (writing in a word processor). Therefore, the effects of recognizing resources from a screenshot in ScreenTrack may not translate well to cases where digital resources in a time-lapse video are not visually distinct, as when switching between text documents. In the future long-term study, we plan to investigate the extent to which computing tasks that include non-visual resources limit the effects of our screenshot-based retrieval method.

7 Conclusion

In this study, we explored the idea of using a visual history for retrieving digital resources such as applications, web pages, and documents. To test the idea, we designed and developed ScreenTrack, and validated it in a series of user studies that explores two different time scales and setups. The result showed that using ScreenTrack led to a productivity gain of 35.9%. We also confirmed that the rich contextual information embedded in the time-lapse videos helps users reconstruct their mental context and facilitates task resumption while moderating potential concerns related to privacy and disk space usage.

8 Acknowledgements

We thank all the reviewers for their thorough feedback. We are also grateful to all participants and Edward Powell for their contributions in the user studies; and Prof. Scott McCrickard and Prof. Denis Gracanin for their advises on this work.


  1. P. D. Adamczyk and B. P. Bailey (2004) If not now, when?: the effects of interruption at different moments within task execution. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, New York, NY, USA, pp. 271–278. External Links: ISBN 1-58113-702-8, Link, Document Cited by: §3.1.
  2. L. Bannon, A. Cypher, S. Greenspan and M. L. Monty (1983) Evaluation and analysis of users’ activity organization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’83, New York, NY, USA, pp. 54–57. External Links: ISBN 0-89791-121-0, Link, Document Cited by: §1.
  3. L. J. Bannon (2006) Forgetting as a feature, not a bug: the dualityof memory and implications for ubiquitous computing. CoDesign 2 (1), pp. 3–15. External Links: Document, Link, Cited by: §4.3.
  4. L. Barkuus and A. Dey (2003) Location-based services for mobile telephony: a study of users’ privacy concerns. In IN 9TH IFIP TC13 INTERNATIONAL CONFERENCE ON HUMAN-COMPUTER INTERACTION, INTERACT 2003, Cited by: §4.3.
  5. M. S. Bernstein, J. Shrager and T. Winograd (2008) Taskposé: exploring fluid boundaries in an associative window visualization. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08, New York, NY, USA, pp. 231–234. External Links: ISBN 978-1-59593-975-3, Link, Document Cited by: §3.2.
  6. T. F. Brady, T. Konkle, G. A. Alvarez and A. Oliva (2008) Visual long-term memory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences 105 (38), pp. 14325–14329. External Links: Document, ISSN 0027-8424, Link, Cited by: §3.3.
  7. G. R. Cangiano and J. D. Hollan (2009) Capturing and restoring the context of everyday work: a case study at a law office. In Human Centered Design, M. Kurosu (Ed.), Berlin, Heidelberg, pp. 945–954. External Links: ISBN 978-3-642-02806-9 Cited by: §3.3.
  8. E. K. Choe, N. B. Lee, B. Lee, W. Pratt and J. A. Kientz (2014) Understanding quantified-selfers’ practices in collecting and exploring personal data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, New York, NY, USA, pp. 1143–1152. External Links: ISBN 978-1-4503-2473-1, Link, Document Cited by: §4.3.
  9. M. G. Christel, A. G. Hauptmann, H. D. Wactlar and T. D. Ng (2002) Collages as dynamic summaries for news video. In Proceedings of the Tenth ACM International Conference on Multimedia, MULTIMEDIA ’02, New York, NY, USA, pp. 561–569. External Links: ISBN 1-58113-620-X, Link, Document Cited by: §6.
  10. M. A. Conway (2009) Episodic memories. Neuropsychologia 47 (11), pp. 2305 – 2313. Note: Episodic Memory and the Brain External Links: ISSN 0028-3932, Document, Link Cited by: §6.
  11. L. Coraggio (1990) Deleterious effects of intermittent interruptions on the task performance of knowledge workers: a laboratory investigation.. Cited by: §5.2.1.
  12. M. Czerwinski, M. Czerwinski, E. Horvitz and S. Wilhite (2004) A diary study of task switching and interruptions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, New York, NY, USA, pp. 175–182. External Links: ISBN 1-58113-702-8, Link, Document Cited by: §1, §3.1.
  13. M. Czerwinski and E. Horvitz (2002) An investigation of memory for daily computing events. In People and Computers XVI - Memorable Yet Invisible, X. Faulkner, J. Finlay and F. Détienne (Eds.), London, pp. 229–245. External Links: ISBN 978-1-4471-0105-5 Cited by: §3.3.
  14. L. Dabbish and R. E. Kraut (2004) Controlling interruptions: awareness displays and social motivation for coordination. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, CSCW ’04, New York, NY, USA, pp. 182–191. External Links: ISBN 1-58113-810-5, Link, Document Cited by: §3.1.
  15. W. Ding, G. Marchionini and D. Soergel (1999) Multimodal surrogates for video browsing. CoRR cs.DL/9902019. External Links: Link Cited by: §4.1.
  16. A. N. Dragunov, T. G. Dietterich, K. Johnsrude, M. McLaughlin, L. Li and J. L. Herlocker (2005) TaskTracer: a desktop environment to support multi-tasking knowledge workers. In Proceedings of the 10th International Conference on Intelligent User Interfaces, IUI ’05, New York, NY, USA, pp. 75–82. External Links: ISBN 1-58113-894-6, Link, Document Cited by: §1, §3.1, §3.2.
  17. D. A. Epstein, B. H. Jacobson, E. Bales, D. W. McDonald and S. A. Munson (2015) From ”nobody cares” to ”way to go!”: a design framework for social sharing in personal informatics. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’15, New York, NY, USA, pp. 1622–1636. External Links: ISBN 978-1-4503-2922-4, Link, Document Cited by: §4.3.
  18. A. Fouse, N. Weibel, E. Hutchins and J. D. Hollan (2011) ChronoViz: a system for supporting navigation of time-coded data. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, New York, NY, USA, pp. 299–304. External Links: ISBN 978-1-4503-0268-5, Link, Document Cited by: §3.3.
  19. E. Freeman and D. Gelernter (1996-03) Lifestreams: a storage model for personal data. SIGMOD Rec. 25 (1), pp. 80–86. External Links: ISSN 0163-5808, Link, Document Cited by: §3.2.
  20. V. M. González, G. Mark and G. Mark (2004) ”Constant, constant, multi-tasking craziness”: managing multiple working spheres. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, New York, NY, USA, pp. 113–120. External Links: ISBN 1-58113-702-8, Link, Document Cited by: §1, §3.1.
  21. T. Grossman, J. Matejka and G. Fitzmaurice (2010) Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST ’10, New York, NY, USA, pp. 143–152. External Links: ISBN 978-1-4503-0271-5, Link, Document Cited by: §3.3.
  22. J. Hailpern, N. Jitkoff, A. Warr, K. Karahalios, R. Sesek and N. Shkrob (2011) YouPivot: improving recall with contextual search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, New York, NY, USA, pp. 1521–1530. External Links: ISBN 978-1-4503-0228-9, Link, Document Cited by: §3.2, §4.2.
  23. L. Hattori, M. D’Ambros, M. Lanza and M. Lungu (2011-06) Software evolution comprehension: replay to the rescue. In 2011 IEEE 19th International Conference on Program Comprehension, Vol. , pp. 161–170. External Links: Document, ISSN Cited by: §3.1.
  24. A. Informatics (2016) ATLAS recall. Note: \url Cited by: §6.
  25. S. T. Iqbal and B. P. Bailey (2007) Understanding and developing models for detecting and differentiating breakpoints during interactive tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, New York, NY, USA, pp. 697–706. External Links: ISBN 978-1-59593-593-9, Link, Document Cited by: §3.1.
  26. S. T. Iqbal and E. Horvitz (2007) Disruption and recovery of computing tasks: field study, analysis, and directions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, New York, NY, USA, pp. 677–686. External Links: ISBN 978-1-59593-593-9, Link, Document Cited by: §1, §3.1, §3.1.
  27. Q. R. Jett and J. M. George (2003) Work interrupted: a closer look at the role of interruptions in organizational life. Academy of Management Review 28 (3), pp. 494–507. External Links: Document, Link, Cited by: §3.1.
  28. S. Jeuris, S. Houben and J. Bardram (2014) Laevo: a temporal desktop interface for integrated knowledge work. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, New York, NY, USA, pp. 679–688. External Links: ISBN 978-1-4503-3069-5, Link, Document Cited by: §3.2.
  29. J. Jo, B. Kim and J. Seo (2015) EyeBookmark: assisting recovery from interruption during reading. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, New York, NY, USA, pp. 2963–2966. External Links: ISBN 978-1-4503-3145-6, Link, Document Cited by: §3.1, §3.1.
  30. S. Kaasten, S. Greenberg and C. Edwards (2002) How people recognise previously seen web pages from titles, urls and thumbnails. In People and Computers XVI-Memorable Yet Invisible, pp. 247–265. Cited by: §1.
  31. A. J. Ko, H. Aung and B. A. Myers (2005) Eliciting design requirements for maintenance-oriented ides: a detailed study of corrective and perfective maintenance tasks. In Proceedings of the 27th International Conference on Software Engineering, ICSE ’05, New York, NY, USA, pp. 126–135. External Links: ISBN 1-58113-963-2, Link, Document Cited by: §3.1.
  32. J. Kopp (1988) Self-monitoring: a literature review of research and practice. In Social Work Research and Abstracts, Vol. 24, pp. 8–20. Cited by: §4.3.
  33. G. Kovacs, Z. Wu and M. S. Bernstein (2018-11) Rotating online behavior change interventions increases effectiveness but also increases attrition. Proc. ACM Hum.-Comput. Interact. 2 (CSCW), pp. 95:1–95:25. External Links: ISSN 2573-0142, Link, Document Cited by: §3.1.
  34. M. G. Lamming and W. M. Newman (1992) Activity-based information retrieval: technology in support of personal memory.. In IFIP Congress (3), Vol. 14, pp. 68–81. Cited by: §3.3.
  35. M. Lamming and M. Flynn (1994) ”Forget-me-not” intimate computing in support of human memory. In , pp. 125–128. Cited by: §3.3.
  36. A. Large, J. Beheshti, A. Breuleux and A. Renaud (1995-06) Multimedia and comprehension: The relationship among text, animation, and captions. Journal of the American Society for Information Science 46 (5), pp. 340–347. External Links: Document, Link Cited by: §4.1.
  37. S. W. Lee and G. Essl (2015) Live writing: asynchronous playback of live coding and writing. In Proceedings of the International Conference on Live Coding, Cited by: §3.3.
  38. D. Lupton (2016) You are your data: self-tracking practices and concepts of data. In Lifelogging: Digital self-tracking and Lifelogging - between disruptive technology and cultural transformation, S. Selke (Ed.), pp. 61–79. External Links: ISBN 978-3-658-13137-1, Document, Link Cited by: §4.3.
  39. G. Mark, G. Mark, V. M. Gonzalez and J. Harris (2005) No task left behind?: examining the nature of fragmented work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’05, New York, NY, USA, pp. 321–330. External Links: ISBN 1-58113-998-5, Link, Document Cited by: §5.2.1.
  40. J. M. J. Murre and J. Dros (2015-07) Replication and analysis of ebbinghaus’ forgetting curve. PLOS ONE 10 (7), pp. 1–23. External Links: Link, Document Cited by: §5.2.1.
  41. G. Neff and D. Nafus (2016) Self-tracking. 1st edition, The MIT Press. External Links: ISBN 0262529122, 9780262529129 Cited by: §4.3.
  42. C. Newport (2019) Digital minimalism: choosing a focused life in a noisy world. Penguin. Cited by: §3.1.
  43. C. Parnin and S. Rugaber (2009-05) Resumption strategies for interrupted programming tasks. In 2009 IEEE 17th International Conference on Program Comprehension, Vol. , pp. 80–89. External Links: Document, ISSN 1092-8138 Cited by: §3.1, §3.1.
  44. C. Parnin and R. DeLine (2010) Evaluating cues for resuming interrupted programming tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, New York, NY, USA, pp. 93–102. External Links: ISBN 978-1-60558-929-9, Link, Document Cited by: §3.1.
  45. L. A. Perlow (1999) The time famine: toward a sociology of work time. Administrative Science Quarterly 44 (1), pp. 57–81. External Links: Document, Link, Cited by: §3.1.
  46. M. Raento and A. Oulasvirta (2008-10) Designing for privacy and self-presentation in social awareness. Personal Ubiquitous Comput. 12 (7), pp. 527–542. External Links: ISSN 1617-4909, Link, Document Cited by: §4.3.
  47. J. Rekimoto (1999) Time-machine computing: a time-centric approach for the information environment. In Proceedings of the 12th Annual ACM Symposium on User Interface Software and Technology, UIST ’99, New York, NY, USA, pp. 45–54. External Links: ISBN 1-58113-075-9, Link, Document Cited by: §3.2.
  48. A. Rule, A. Tabard, K. Boyd and J. Hollan (2015-07) Restoring the Context of Interrupted Work with Desktop Thumbnails. In 37th Annual Meeting of the Cognitive Science Society, Pasadena, United States, pp. 1–6. External Links: Link Cited by: §1.
  49. A. Rule, A. Tabard and J. Hollan (2017-11) Using visual histories to reconstruct the mental context of suspended activities. Hum.-Comput. Interact. 32 (5-6), pp. 511–558. External Links: ISSN 0737-0024, Link, Document Cited by: §1, §3.3, §4.1, §4.1, §4.2, §4.3, §4.4.
  50. D. M. Russell and M. Oren (2009-01) Retrospective cued recall: a method for accurately recalling previous user behaviors. In 2009 42nd Hawaii International Conference on System Sciences, Vol. , pp. 1–9. External Links: Document, ISSN 1530-1605 Cited by: §1, §3.3, §4.1.
  51. D. M. Russell and E. H. Chi (2014) Looking back: retrospective study methods for hci. In Ways of Knowing in HCI, J. S. Olson and W. A. Kellogg (Eds.), pp. 373–393. External Links: ISBN 978-1-4939-0378-8, Document, Link Cited by: §1, §3.3, §4.1.
  52. I. Safer and G. C. Murphy (2007) Comparing episodic and semantic interfaces for task boundary identification. In Proceedings of the 2007 Conference of the Center for Advanced Studies on Collaborative Research, CASCON ’07, Riverton, NJ, USA, pp. 229–243. External Links: Link, Document Cited by: §3.3.
  53. D. D. Salvucci (2010) On reconstruction of task context after interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, New York, NY, USA, pp. 89–92. External Links: ISBN 978-1-60558-929-9, Link, Document Cited by: §5.2.2, §5.2.2.
  54. S. D. Scott, S. Mercier, M. L. Cummings and E. Wang (2006) Assisting interruption recovery in supervisory control of multiple uavs. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50 (5), pp. 699–703. External Links: Document, Link, Cited by: §3.1.
  55. A. J. Sellen, A. Fogg, M. Aitken, S. Hodges, C. Rother and K. Wood (2007) Do life-logging technologies support memory for the past? an experimental study using sensecam. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, New York, NY, USA, pp. 81–90. External Links: ISBN 9781595935939, Link, Document Cited by: §3.3.
  56. G. Smith, P. Baudisch, G. Robertson, M. Czerwinski, B. Meyers, D. Robbins and D. Andrews (2003) Groupbar: the taskbar evolved. In Proceedings of OZCHI, Vol. 3, pp. 10. Cited by: §3.1, §3.2.
  57. L. Standing, J. Conezio and R. N. Haber (1970-08-01) Perception and memory for pictures: single-trial learning of 2500 visual stimuli. Psychonomic Science 19 (2), pp. 73–74. External Links: ISSN 0033-3131, Document, Link Cited by: §3.3.
  58. B. Xu, P. Chang, C. L. Welker, N. N. Bazarova and D. Cosley (2016) Automatic archiving versus default deletion: what snapchat tells us about ephemerality in design. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ’16, New York, NY, USA, pp. 1662–1675. External Links: ISBN 978-1-4503-3592-8, Link, Document Cited by: §4.3.
  59. Y. Yoon, B. A. Myers and S. Koo (2013-Sep.) Visualization of fine-grained code change history. In 2013 IEEE Symposium on Visual Languages and Human Centric Computing, Vol. , pp. 119–126. External Links: Document, ISSN Cited by: §3.1, §3.3.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description