Anteater: Interactive Visualization for Program Understanding

Anteater: Interactive Visualization for Program Understanding

Abstract

Understanding and debugging long, complex programs can be extremely difficult; it often includes significant, manual program instrumentation and searches through source files. In this paper, we present Anteater, an interactive visualization system for tracing and exploring the execution of a program. While existing debugging tools often have visualization components, these components are often added on top of an existing environment. In Anteater, in contrast, visualization is a driving concern. We answer the following question: what should a debugging tool look like if it were designed from the ground up to support interactive visualization principles? Anteater automatically instruments source code to capture execution behavior along with variables and expressions specified by the user. After generating the execution trace, Anteater presents the execution information through interactive visualizations. Anteater supports interactions that help with tasks such as discovering important structures in the execution, learning dependencies, and understanding and debugging unexpected behaviors. To assess the utility of Anteater, we present several real-world case studies that show how Anteater compares favorably to existing approaches. Finally, we discuss limitations of our system and where further research is needed.

Interactive Visualization, Program Traces
\onlineid

1104 \vgtccategoryResearch \vgtcpapertypesystem \authorfooter Rebecca Faust, Katherine Isaacs, and Carlos Scheidegger are with the HDC Lab, Department of Computer Science, University of Arizona. E-mail: {rjfaust, kisaacs, cscheid}@email.arizona.edu William Z. Bernstein and Michael Sharp are with NIST. E-mail: {wzb, michael.sharp}@nist.gov \shortauthortitleFaust et al.: Anteater \CCScatlist \CCScatK.6.1Management of Computing and Information SystemsProject and People ManagementLife Cycle; \CCScatK.7.mThe Computing ProfessionMiscellaneousEthics \teaser A programmer investigates a bug in their code. One common practice, shown in the top row, is to instrument the program manually to collect suspicious variables (in this case ), and print out their values. Manual instrumentation, however, is itself error-prone and leaves the programmer responsible for sifting through values. Another common practice, shown in the second row, is to use a debugger to stop the execution of the program automatically and view each assignment of individually. However, breakpoint-debugging tends to provide only a narrow, one-at-a-time view of the values. Anteater, shown in the bottom row, automatically instruments the code to track variables along with the context of their execution structure. It presents the programmer with interactive visualizations that give a global view of values, which allow for easy detection of erroneous values as well as interactions to narrow down the visualizations to inspect specific values. We provide a demo of Anteater in action as supplemental material. \vgtcinsertpkg

\firstsection

Introduction

Visualization practitioners now have a well-defined set of principles to drive the design, development, and testing of interactive visualization software [6, 10, 36]. Such principles have been applied in a variety of domains [22, 42, 29]. We argue that computer-program comprehension and debugging, despite its importance and difficulty, has received comparatively little attention. Concretely, we tackle the following question: what should a system for debugging programs look like when interactive visualization principles provide the driving vision?

Current debugging strategies are tedious and time consuming, often involving stepping through debuggers, adding logging statements, or searching through source code - either manually or with a code browsing tool [26]. Traditional debuggers require programmers to set breakpoints at which they inspect the code’s execution by stepping through its line-by-line operation. Tiarks et al. [40] observed that programmers experience difficulties in choosing breakpoint locations and quickly forget analysis details while navigating across the code, a common activity during debugging. Furthermore, programmers can only access a single instance of a variable’s value at a time; they are not offered a global view of a variable’s behavior. To obtain an overview of the behavior of a critical variable, programmers instrument their code to back out value instances, typically by generating print statements or writing to a log. Once programmers understand a variable’s behavior, they remove the instrumentation, often only to put it back later to investigate another critical variable’s behavior. We see a need for an exploratory debugging solution that provides more effective global views of values and reduces the cognitive burden on programmers.

Understanding and making sense of unfamiliar code is another common problem users face when programming [40]. Whether it be code they pulled from the internet, a program handed off to them by a collaborator, or an existing project that they have joined, sifting through the program to understand its execution and determine where to start in modifying it proves challenging. Something as simple as identifying the dependencies of a variable can become a significant burden when the program is large. There is a need for facilitating a deeper understanding of the structure of a program’s execution to assist programmers in exploring how functions and variables depend on one another.

In response to these challenges, we present Anteater, a system for debugging and understanding programs designed using the principles of interactive visualization. In taking a visualization-first approach, Anteater provides more informative overviews of a program’s behavior while supporting interaction to dig deeper into the details of the execution. It reduces the amount of effort required from a user by 1) instrumenting programs to collect the values they want to inspect automatically and 2) allowing them to zero in on values of interest easily without stepping through a debugger.

In this paper, we present a prototype implementation that traces a program to capture not only the execution structure but also values of interest in context of the execution. Anteater then presents this trace to the user through well-understood visualizations. Anteater’s interactions facilitates insight into a program’s execution. In summary, our contributions include the following:

  • a goals-and-tasks analysis [25] of the typical practice of program debugging (Section 3)

  • a description and prototype implementation of Anteater, aimed at providing first-class interactive-visualization support to program debugging and understanding

  • three case studies of real-world programs that demonstrate how our system compares favorably to existing approaches

1 Related Work

Visual Debugging Many attempts have been made to leverage visualization principles to augment the debugging process. Some of these efforts focus on adding visualization options to breakpoint and step-through debuggers [11, 27, 13, 32, 9, 34, 28]. Other visualizations aim at showing the user-specific information about the execution, such as an overview of the heap [3], the impact of resource utilization on control flow [30], object mutation [35], or run-time state and data structures of the program [39].

A commonality across nearly all of these visual-debugging tools is the lack of global views of values from variables and expressions within a program’s execution. Some tools attempt to provide additional context by allowing back-stepping in the debugger or providing a history of the execution  [32, 16, 27]. Aftandilian et al. [3] give a global view of the heap by taking snapshots throughout the program. Schulz et al. [35] provide a global view of object mutations; if the object is numeric, the global view shows the value behavior throughout the execution. Some tools also attempt to give global views of value behaviors by introducing sparklines, plotting the values in the original line of source code [5, 19].

Anteater differs from these works by displaying global views of variables and expression values in the context of the execution. This new perspective can be key to understanding why certain values occurred and providing more visualization options and interactions to facilitate exploration.

Alsallakh et al. [4] created an Eclipse1 plugin that tracks specific tracepoints (equivalent to a breakpoint in a debugger) throughout a program’s execution. Watchpoints can also be added to a field on which the tracer will track assignments. The tracepoint instances are visualized as individual line charts where interactions provide additional information about the program at that point and watchpoints are viewed as a step chart of the values over time. While the goals of this plugin closely relate to those of Anteater, Anteater stands apart for two reasons. First, it traces all calls and loops, rather than user-defined tracepoints, along with the values desired by the user. Second, it presents all this information in a trace visualization with corresponding plots of the tracked values. This information can provide the context necessary to better understand why variables take on certain values. Furthermore, the plugin is Eclipse-dependent whereas Anteater generalizes to any language that supports transforming programs in the way Anteater does (including C++  [1] and Javascript [18]).

The most similar tool to Anteater is Kang et al.’s [23] Omnicode. Omnicode provides run-time visualizations of program states. It aims to help novice users build mental models about programs and debug their code through a live coding environment that updates variable visualizations in real time. The primary visualization provided is a scatterplot matrix displaying plots for each variable over execution steps. Omnicode also provides plots of each variable against each other, and, for a specific step in the execution, a visualizations of the heap. While Omnicode and Anteater have much in common, they were designed for different audiences (novices vs. general programmers) and thus support different types of programs. Similarly, the supported visualizations are geared toward the types of tasks encountered by their target audiences. The comparison between Omnicode and Anteater is discussed in more depth in Section 7.

Trace Visualization People have tried to provide assistance in program understanding by visualizing program traces. Many of these visualizations are designed for view traces of parallel programs [37, 24, 41]. There also exists a significant body of trace visualizations that use icicle plots and flame graphs as the primary visualization  [33, 24, 17, 7, 41]. Anteater presents its trace in a plot similar to icicle plots and flame graphs because these plots are common and well-understood for visualizing execution traces. However, Anteater differs in its definition of trace. While these previous traces capture the calling structure of the execution, Anteater extends this to capture values of marked variables and expressions as well as loop behaviors to provide a more detailed execution trace. This extension provides users with additional context for how values are reached. 3

Figure 1: An overview of Anteater on a recursive Fibonacci program, tracking the variable “val”. (A) shows the initial view presented by Anteater. The generalized context tree, or icicle plot, shows the execution structure. The teal blocks represent function calls while the varying shades of purple represent the value of “val” at that instance. We can see the recursive calling structure of the Fibonacci function and can easily identify where it is repeating work. When a block is clicked on, its dependencies are highlighted in red. We can see that the selected block, which is an instance of “val”, depends on the prior two calls to the Fibonacci function. The histogram shows the distribution of the values of the “val” variable to give an overview of its behavior. We can see that the majority of values are in the 1-5 range. In (B), we zoom in on a block to get a closer look at the structure that occurs within that call. To help us remember our position in the generalized context tree, a grayscale copy is shown below highlighting the region of interest. We have switched to the scatterplot view of the variable “val” over time. Brushing over the scatterplots highlights the corresponding instances in the generalized context tree and the context bar. From the scatterplot, we can see repetitive patterns that indicate that Fibonacci is doing redundant work.

2 Background

In this section, we discuss the current state of program debugging, understanding, and tracing, as well as the need for a system like Anteater.

2.1 Understanding Programs

The situation frequently arises where a programmer needs to understand code they did not write. Navigating unfamiliar source code is not an easy task and there are no tools designed specifically to facilitate such understanding. Several articles and blog posts exist to help programmers effectively read source code [12, 20]. Much of this help, however, consists of generic advice only. One article advises programmers to run the program first then find something the code is doing and trace it backwards. Another advises to use a debugger to set breakpoints and find a tool that allows for more intelligent navigation through the source code, such as Sourcegraph [38] and Pycharm [21].

From these posts, it seems that a programmer’s best tools for understanding unfamiliar code are debuggers, print statements, and tools for source code navigation. With only these tools to help understand code, programmers must expend considerable effort to carry out several burdensome tasks. Those tasks include 1) finding a starting point for navigation, 2) setting breakpoints to step through debuggers, and 3) making small changes to the program to better understand the impact on the execution.

2.2 Debugging Scenario

Programmer Patty has a bug in her code. She believes that the bug is occurring in a specific loop but cannot identify the root cause. Using a typical debugger, she sets a breakpoint at the beginning of the loop and runs the debugger. When the debugger reaches the breakpoint, she inspects values and takes a few steps through execution but does not yet see the bug. Patty continues the program until it hits the breakpoint again at the next iteration. She continues to step through each iteration of the loop but has little success in finding the bug.

After several iterations, Patty gives up on using the debugger and modifies the code with print statements. She prints the variable she believes is causing the bug and runs the program. Patty scans through the printed values, trying to find any erroneous values; but, her loop has many iterations and she quickly gets lost in the print statements.

Her next idea is to write the values to a file and plot them. Patty first alters her source code to write the values to a file. She then writes a script that reads the file and plots the values. Now she can see the behavior of every instance of the value and pinpoint the incorrect values. With this information, Patty returns to the debugger and stops the program when it reaches the iteration containing incorrect values to find the root cause.

The scenario described above encompasses the typical ways programmers debug their programs [40]. While not every bug requires all of these methods, programmers typically use more than one of them. The fact that many programmers use a combination of independent debugging-methods when fixing their programs prompts the question: can we design a better debugger that 1) reduces the amount of manual instrumentation required, 2) gives the users greater control over the values they see, and 3) provides them with a visualization option automatically? While various debugging tools address aspects of these problems, no debugger comprehensively addresses all of them. There exist tools that add visualizations on top of existing debuggers, such as Mirur [9]. However, many of them still operate within a snapshot of the program. Rather than visualizing the global behaviors of values, such tools help visualize more complex objects at a given timestep.

Figure 2: An overview of the Anteater system. First a user chooses variables to track, defining the trace specification, using the Anteater interface. This trace specification is sent through the web interface to the python back end, along with the source code. The Anteater tracer then instruments the source code to collect execution information along with the specified values. The right side of the figure shows a simplified version of this instrumentation. After the code is instrumented, it is run using python to create the program trace. This trace is passed back through the web interface to the Anteater front end where it is visualized and presented to the user.

We designed Anteater to address these problems in a more comprehensive way, by using the principles of interactive visualization. Fig. Anteater: Interactive Visualization for Program Understanding gives an overview of how Anteater compares to standard debugging practices. Rather than presenting users a snapshot of everything at a single timestep, we present them with global views of targeted variables of interest with interactions to narrow down to specific values. In doing this, users can more easily discover patterns within the values they deem important. We pair these global views with a visualization of the execution structure that allows users to maintain context with the execution and inspect execution information surrounding the values. Fig. 1 shows an overview of Anteater.

If Programmer Patty had been using Anteater, she could have easily set Anteater to track the value she believed to be raising issues along with any other values that she believed to be candidate roots of causation. Anteater would then trace her program and provide her with visualizations that would allow her to identify the iterations where the value was incorrect. Patty could then filter down the execution visualization to inspect the rest of the values she tracked at those iterations. With Anteater, Patty is able to complete all of her debugging in one place using only a few interactions and requiring no manual instrumentation.

2.3 Tracing

Historically, program traces have captured performance information for a program’s execution. They typically track function call and time spent within a function. Visualizations are then created to present this information to users. However, function calls and execution time only solve a subset of bugs. Primarily, program traces help identify functions that are taking more time than necessary or call structures that are not expected. They do little to help with finding bugs in the actual values being calculated by the program. Anteater expands the definition of a trace to include values from within the program. Capturing these values in the context of the trace, rather than collecting them in a separate file, provides important context to the programmer. A context that is lost when looking at the values in isolation.

3 Task Analysis

In this section, we discuss Anteater’s goals that were selected based on both current standards of practice from literature and our own experiences with respect to program debugging and understanding.

G1: Identifying and understanding the source of unexpected execution behavior When programmers write and execute programs, they have some expectation of what their program should be doing and what parts should be executing. As a result, one goal of debugging is to identify what is causing an execution to deviate from what the programmer expected.

G2: Identifying and understanding the source of unexpected values and trends Similar to G1, programmers typically have ideas about what values they should observe during the execution; and, thus, they desire to identify the root cause of unexpected values in the execution.

G3: Understanding an unfamiliar piece of code Frequently, programmers are tasked with understanding code written by someone else. Typically, this is no easy task and requires a significant amount of effort on the part of the programmer.

Under the framework of Lam et al. [25], the identifying portions of the goals G1 and G2, along with the entirety of G3, fall into the “Discover Observation” category. The understanding portions of G1 and G2, however, fall into the “Identify Main Cause” category. From these goals we derived several sub-tasks.

T1: Track a variable or expression It is often useful to look at the values that a variable or expression take on to determine if it is behaving as-expected and to identify any erroneous values. This task supports G2 and G3.

T2: Identify what functions are called at runtime Often it is not clear from the static source code which functions will execute and when. However, identifying which functions are actually called during an execution is crucial for understanding how a program is operating. This task supports G1 and G3.

T3: Identify dependencies for a variable Understanding dependencies is crucial when trying to understand unfamiliar code. Identifying how a value is calculated, including the execution path required to complete the variable’s calculation, allows programmers to better understand the underlying nature of the value in question. Such insight can lead to finding the cause of an unexpected value. This task supports G1 and G3.

T4: Identify interesting subsets of values Given a variable or expression, it is important to be able to identify the subset of values that correspond to interesting behavior. For example, if certain values indicate a failure in the program, they need to be identified so the surrounding values can be examined to understand the cause of the behavior. This task supports G1 and G3.

T5: Observe relationships between values When debugging a program, programmers often investigate relationships between variables. For example, if variable changes, how does variable change? These relationships may not be explicitly defined by the code, i.e., may not directly depend on . Uncovering such relationships contributes to program understanding. This task supports G1 and G3.

T6: Maintain context between runtime state and static source When trying to debug and understand a program, maintaining context with the actual code is critical. If the programmer is manually instrumenting print statements, they also must codify contextual information to derive insight, e.g., representing the location of a variable’s modification. This task supports G1, G2, and G3.

A system that supports all of these tasks needs to be able to track the execution structure of the program along with variable and expression values in context of this execution. An execution trace is a natural fit for tracking the execution structure and can be modified to also collect values. Once this data is collected, it needs to be presented in a way that allows for easy navigation through it while supporting these tasks. We argue that visualization is the best way to present this information because it is known for providing overviews and context, highlighting relationships, and facilitating filtering down to subsets of interesting information, all of which are needed to support these tasks. Anteater takes a visualization approach to program debugging and understanding that satisfies these goals through execution traces and visualizations. Anteater deals solely with single-threaded programs but we expect that this task analysis would need to be extended to satisfy our goals for multi-threaded programs.

4 Tracing Infrastructure of Anteater

To support the goals and tasks defined in Sec. 3, an execution trace with accompanying variable and expression values must be collected. Anteater implements a tracer that automatically instruments the source code to collect the execution trace. Implemented in Python, the tracer relies solely on the Abstract Syntax Trees (AST) library to facilitate the transformation of the source code. While Anteater currently only works with Python programs, the same principles can be implemented in any language that has the ability to transform source code in a similar way. Fig. 2 illustrates how the system operates.

4.1 Tracing Programs

When a user chooses to create a trace, the Anteater backend is passed a trace specification containing a list of variables and expressions to track along with a list of functions and libraries to exclude from the trace (this is discussed more in Section 5). The tracer goes through each of these lists and determines the scope in which each item resides to ensure that it only tracks/excludes the specified items. For example, if two functions both define variable , the tracer will only track the one the user selected.

Once the scope of each item is determined, the tracer uses the Python ast library to parse the source code into its AST. What follows is a series of traversals of the AST to collect information about the source code and transform the program to trace the execution.

In the first traversal through the AST, no transformations occur but rather information about functions, loops, and dependencies is collected. For functions and loops, it collects the lines at which the function definition or loop begins and ends. This information enables more detailed linking between the visualizations and the source code. For dependencies, the tracer traverses through the source and for each variable stores the functions and variables that it directly depends on (i.e., that are on the right side of an assignment statement). To find all dependencies for a given variable, we access its dependency list; and, for each dependency in the list, we access their dependencies. This continues until a comprehensive list of all possible dependencies is built.

Once all of the static data has been retrieved from the source code, Anteater begins transforming it. An initial traversal through the AST transforms the code to separate all function calls from their respective expression statements. All function calls that are not stand alone statements are pulled out of their expressions and assigned to a temporary variable that then replaces the call in the original expression (e.g., becomes ; ). This will allow Anteater to capture easily when functions are called and in what order.

Now the tracer does the main transformations to insert the instrumentation that collects the trace. As the tracer traverses the AST, it always pauses at assignment, call, and loop nodes. When it reaches an assignment node, it checks if the target variable needs to be tracked. If so, it inserts new nodes into the AST that record the value of the variable after assignment.

When the tracer reaches a call node, it first checks if the call is in the exclusion list. If it is not, the tracer generates AST nodes to record that the program is about to enter the call and inserts them before the call. The tracer then generates AST nodes that record when the call has returned and inserts them after the call. Because we moved function calls into their own statements, a function call statement is fully executed before the next function call starts. This allows all bookkeeping for a call to be completed before the next call executes. A simplified example of this transformation is shown in Fig. 2

When the tracer reaches a loop, it creates a counter that counts the iterations of that loop and inserts new instrumentation to record that it has entered a loop. As it traverses the body of the loop, any time a new record is created, the tracer records the iteration in which that record occurred. Tracking the iteration binds together groups of records in the trace; records that occurred in the same part of the execution.

Last, the tracer transforms the program to record expressions. Expressions are more complicated because they could occur in a variety of nodes. As the tracer visits the nodes, it checks if the line containing the node also contains a tracked expression. If it does, the tracer extracts the expression from the line, assigns it to a temporary variable, and then replaces the expression in the original line with the temporary variable. This ensures that the expression only executes once and that the trace records its exact behavior during the execution of the program.

Once all of the instrumentation is in place, Anteater complies the AST into an executable program, which generates the trace as it executes.

4.2 Trace File Structure

The trace files themselves are JSON files. JSON allows us to record the hierarchical structure of a program execution easily and pass the trace to the Javascript front end without reformatting. When the tracer reaches a function call or loop, it writes JSON to represent that block along with a “body” field to hold records that occur within its body.

5 Anteater’s Visualization Design

Anteater presents a new way of exploring and interacting with program executions helping users get a deeper understanding of the inner-workings of their programs that they cannot get from traditional tools. In the previous section, we discussed how Anteater creates the execution trace. Here, we describe the visualization design of Anteater and the features that facilitate the exploration of the execution trace. As we walk through the design, we will describe the features in context of a simple Python program that runs a recursive Fibonacci function.

5.1 Creating a Program Trace

To fulfill T1, Anteater needs to allow a user to define what variables and expressions to trace. The first page a user sees after they load their source allows them to set the specifications of the trace. This page is shown in Fig. 3. The user can highlight text in their source code and right click to specify the action (track or exclude) for that text. Once the users completes the trace definition, they click “Run Trace,” which passes the trace specification to the tracer on the back end for processing.

The tracer will only collect variables and expressions that the user specifies. This was an explicit design choice because the entirety of data associated with every single variable is massive. Collecting all variables would also record unnecessary data. Many variables residing in a program have little importance in describing how code is behaving. Thus, we allow the user to select the important variables to track.

Similarly, collecting all function calls leads to a large collection of unimportant information. For example, it is most likely not important to know every time print() is called. To help reduce clutter from unimportant function calls, we add a predefined list of functions and libraries to ignore (e.g., Math, numpy, print, len) and allow users to add functions of their own to exclude from the trace. This gives users the opportunity to reduce clutter in their traces and remove uninteresting/unimportant structures from the execution to better highlight the important structures.

Figure 3: Interface to define a trace. Users highlight variables and expressions in the source code, then right-click to track them. Users can highlight function calls and library imports to exclude them from the trace.

5.2 Visualizing Trace Data

Once the tracer returns the execution trace, Anteater generates interactive visualizations. Two types of visualizations are provided: a view of the execution structure, which we call the generalized context tree after Boehme et al. [8], and a visualization of the variable values.

Generalized Context Tree

The generalized context tree, shown on the right side of Fig. 1-A, provides an overview of the execution structure. The visualization has its origins in flame graphs and icicle plots. We chose this type of visualization because it is well known for visualizing traces and well understood. In our setting, each rectangular block in the plot represents one of three things: a function call, a loop, or a variable assignment. The icicle plot shows the hierarchy so that for a given block, everything that falls below it, that is within its bounds, is a child. For example, in Fig. 1-A the block in the second row labeled “10: val = ” is the initial call into the Fibonacci function and everything below that happens within that call. The generalized context tree can be used to determine which functions executed and when, fulfilling T2.

As we move from left to right in the plot, we are increasing in time; everything to the left of a block was executed before that block. This allows users to easily read the visualization and understand when blocks are executed relative to other blocks.

In the generalized context tree, a single variable is highlighted, selected by the user (in the upper right corner of Fig. 1-A). When they select a variable, all blocks in the tree corresponding to the variable (which reside at leaf level) are colored by the value of that instance. Positive values range from white (low) to purple (high), while negative values range from white (least negative) to orange (most negative). In Fig. 1-A, Anteater colored the leaf nodes representing the variable ”val” with varying shades of purple. Deeper leaves are shaded much lighter, which indicates small values at those instances; this corresponds to the deepest Fibonacci calls returning the smallest values. Coloring blocks in this way shows the behavior of values in the context of the whole execution. Every other variable or expression that appears in the trace still appears in the generalized context tree; Anteater colors them gray.

Figure 4: Debugging Gradient Descent with Anteater. In (A) it is immediately apparent in both the generalized context tree and the histogram that there is a bug causing NaN’s. In (B), we switch to the scatterplot view to see how the values behave before they become NaN. The values are mostly centered around zero before becoming an extremely small negative, then going to infinity and becoming NaN. We suspect that the values centered around zero are not actually zeros so we partition the loop to allow us to zoom in on them and switch to a symmetric log scale,shown in (C). Now we see that the values are oscillating which suggests the problem of exploding gradients caused by a training rate that is too large. In (D) we change the gradient and see that the value quickly converges as it should.

Variable Value Plots

The second visualization provided by Anteater, is a plot of tracked variables. Initially, Anteater shows a histogram of user-selected variables or expressions in the Generalized Context Tree, as seen in the plot in Fig. 1-A. This presents an overview of the distribution of values that the variable or expression takes on throughout the execution. From the histogram in Fig. 1-A, we see that the majority of the values calculated are small (between 1 and 5). Anteater can handle cases where the values take on not a number (), and by creating special bars to represent these values, as seen in Fig. 4.

Users can switch between different plotting views by clicking on the icons above the plot. The first icon, selected as default, is the histogram. Next to the histogram is a scatterplot view. Once clicked, the scatterplot view will show the current variable plotted over time to provide an overview of the value behavior, as shown in Fig. 1-B. To support T5, Anteater provides the option of changing the second variable shown from time to some other compatible variable, allowing the user to observe their relationship. A variable is compatible if it has the same number of instances as the current variable, allowing users to view relationships between variables.

However, by plotting two variables against each other we lose the context of time. Interactions (described in the next section) help bring back some of this context. Anteater also provides a connected scatterplot option (the third plot icon) that will connect and color the points by time, as shown in Fig. 7. When plotting two variables against each other, users may want to flip the plot axes to get a different view of the relationship; Anteater provides an option to do this easily. The last plotting option provided is control over the type of y-scale. If a linear scale is not appropriate for the plot, users can change the y-axis to be a symmetric log scale to better fit their data.

5.3 Interacting with the Trace Visualizations

Anteater’s interactions are key in helping users get a better understanding of their program. Anteater offers several interactions that allow the user the capabilities to 1) maintain context with the source code, 2) inspect the dependencies of a certain instance of a variable or expression, 3) link between the value plots and the generalized context tree, and 4) filter the visualizations to narrow the focus to an interesting/important subset. The last capability enables users to better understand the relationship between specific values and the structure of the execution.

Maintaining Source Code Context

When exploring the execution it is important to link back to the source code to maintain the context of the execution. On its own, the generalized context tree is fairly abstract. To provide necessary context, we pair it with interactions so that when a block in the generalized context tree is selected, it jumps to, and highlights, the corresponding section of the source as shown in the code view of Fig. 1-A. If it corresponds to a user-defined function call, it also highlights the corresponding function. This interaction, paired with a preview of the corresponding source code on the blocks, supports T6 by allowing users to navigate the execution trace without forgetting their place in the source code.

Inspecting Dependencies

To support T3, Anteater determines what dependencies could exist for any instance of a variable (as discussed earlier) and then uses context from the execution trace to eliminate some possibilities and present the remainder to the user. When a user selects a block representing a variable in the generalized context tree, Anteater checks if the prior siblings as well as the siblings of any ancestors of that block are in the list of possible dependencies. If they are, their blocks are highlighted in red to show the user which parts of the context tree that block depends on. This allows the user to quickly get an idea of what is contributing to that specific instance. In Fig. 1-A, we see that the selected instance of “val” depends on the prior two calls to “fib”.

Linking visualizations

Anteater provides interactions on the plots and the generalized context tree to link the two together. When a block is selected in the generalized context tree, the values shown in the plot are filtered down to include all values in the subtree rooted at the selected block. In addition, to provide global context, the plot shows the values from the subtree rooted at the parent of the selected block. As shown in the histogram in Fig. 1-A, the bar that represents the selected instance(s) is shown in black while the rest of the bars are shown in gray for context. In the scatterplot, the points representing selected instances are black while the rest are gray. We also provide linking from the plot back to the generalized context tree. If the plot is a histogram, selecting a bar will highlight the corresponding blocks in the tree. If the plot is a scatterplot, brushing over a set of points will highlight the corresponding blocks in the trees. An example of brushing the scatterplot is shown in Fig. 1-B where the red blocks in the tree correspond to the brushed points. These interactions support T4 by allowing the user to pinpoint interesting values in the plots and easily locate them in the execution.

Filtering

Anteater supports a few types of filtering on the plot and the generalized context tree to help the user filter out unimportant information and emphasize important parts of the execution, which helps support T4. The first type of filtering was mentioned above where clicking on deeper nodes in the context tree filters the value plots. By doing this, users can filter down the plot to subsets of the data that they desire to see. However, there are often loops with many instances of a variable where users desire to filter out some of the values. To help users filter down values, Anteater gives users the ability to partition loops and add new levels in the hierarchy.

When a user is viewing the scatterplot of a variable over time, they are able to brush over a subset of the points, right click, and choose to partition the values around that subset. Anteater then determines which iteration the first brushed value occurred in, which iteration the last brushed value occurred in, and then partitions the loop around those iterations. These new blocks show up as yellow blocks in the generalized context tree. Now the user can filter down the visualization to view only the instances that occur in those iterations. An example of this can be seen in Fig. 4(C).

One last way users can filter the visualization is by hiding parts of the generalized context tree. Right clicking on a block in the tree will expand the block to take up the entire width which increases the size of all of its children, thus making them easier to see. However, in doing this, we will lose context of where we are in the execution. To retain this context, we add a smaller, grayscale version of the generalized context tree with a highlighter bar over it. When the user zooms in on a block, the highlighter narrows to indicate its place in the overall context tree. It will also highlight the selected block in yellow as well as any blocks that are highlighted in the generalized context tree (dependencies and brushed values) This allows users to see them even if they are outside of the visible portion of the generalized context tree. The generalized context tree in Fig. 1-B is zoomed in on the third block, and we see in the context bar our location in the execution.

Figure 5: Anteater views of the num_intersections variable. (A) shows an overview of the behavior of 6 gradient descent methods that should be minimizing this variable. The scatterplot shows that the last three methods get stuck immediately. In (B), vanilla gradient descent initially moves towards a poor solution (a large value) but at the end decreases to a better value. (C) shows how in the momentum method, the number of intersections jumps to a high value every time it descends to a low, potentially good solution. Similarly, in the Nesterov method shown in (D), the solution will decrease for a while before jumping up to a high value and repeating. In the end, vanilla gradient descent performs the best.

6 Case Studies

In this section, we present several, real-world cases, showcasing how Anteater derives insight into debugging and program understanding.

6.1 Gradient Descent

The first case study we present inspects a program performing gradient descent. This program was collected from a question on Stack Overflow [2]. The programmer struggled to figure out why the resulting values were NaNs. We will walk through how to use Anteater to understand the bug and correct it.

First, we run the program with Anteater to track the misbehaving variable “.” Fig. 4-A shows the resulting execution trace and histogram of values. We see from the histogram that much of the descent generates NaNs.

As a natural next step, we look at these values over time. We switch the plot type to “scatterplot” which shows a plot of the variable “” over time, shown in Fig. 4-B. Now we clearly see that the value of “” stays around zero, before becoming a very small negative, then going to infinity after which we hit the NaNs. However, there is something strange in the values staying around zero and then suddenly becoming a very small negative. To investigate this, we partition the loop into two groups: (1) the points staying close to zero and (2) the extremely negative value, infinity and NaN’s. We zoom in on the group containg the points close to zero. We also switch to a symmetric log scale because we suspect that the values may not actually lie that close to zero. The resulting visualizations are shown in Fig. 4-C. We see that the value oscillates between increasingly large positives and negatives until it reaches infinity.

Now that we know the problem, we try to fix it. The oscillating values suggest that the gradients is exploding due to a training rate that is too large. In Fig. 4-D, after lowering the training rate and re-running the trace, the value quickly converges, as expected.

Using Anteater, we can quickly and easily track the variable “” and see its behavior throughout the execution. In a traditional debugger, detecting this behavior would have required stepping through several iterations to view the values. After lowering the training rate, we repeat this process to determine if that fixed the problem. This involves significantly more interaction with the debugger than when using Anteater.

6.2 Graph Edge Crossing Angle Maximization

In this case study, we investigate a program tries to balance the number of edge crossings in a graph with the size of the minimum crossing angle. The program searches for the layout that minimizes the number of edge crossings while maximizing the size of the minimum crossing angle. Researchers in this space test out several gradient-descent methods to determine which one best balances these two costs. In this case study, we use Anteater to compare all of the different gradient-descent methods as well as investigate the differences between multiple runs of the best performing gradient-descent method.

Figure 6: Anteater views of the minimum crossing angle variable. (A) shows an overview of the behavior of 6 gradient descent methods that should be maximizing this variable. The scatterplot shows that the last three methods either get stuck immediately or only slightly increase. (B) shows that the vanilla gradient descent starts off poorly but then finishes at a good value. The momentum method shown in (C), exhibits a pattern, i.e., every time it hits a high, potentially good value, it quickly begins decreasing. Similarly, the Nesterov method shown in (D) drops every time it hits a high value and then begins increasing again. In the end, for minimum crossing angle, vanilla gradient descent performs the best.

Comparing Gradient Descent Methods

We first look at the behavior of all six gradient-descent methods to determine which one perfroms best. The six methods considered are vanilla, momentum, Nesterov, Adagrad, Rmsprop, and Adam. Anteater tracks the number of intersections and minimum crossing angle, which change in each iteration of a given gradient descent method. The resulting visualizations of the number of intersections and the minimum angle are shown in Figs. 5 and 6, respectively. From Fig. 5-A we immediately see the last three methods (Adagrad, Rmsprop, and Adam) do not improve on the number of edge crossings. Fig. 6-A shows that the minimum angle behaves similarly. These three options are ruled out because they do not seem to actually optimize the parameters, but rather stay at the values where they started.

After ruling the last three out, we take a closer look at the other three, first considering the vanilla gradient descent, shown in Fig. 5-B and Fig. 6-B. In Fig. 5-B, the number of intersections initially decreases (as desired) but quickly starts to increase. It stays high for several iterations before dropping off at the very end, providing a good result. Similarly, in  6-B, we see that initially the minimum crossing angle is high (as desired) but quickly drops and stays low until the very end when it spikes back up. This suggests that the number of intersections and minimum crossing angle are correlated; in general, a high minimum angle corresponds to a low number of intersections. Overall, the vanilla gradient descent does not get progressively better results throughout the descent but still finds a good result in the end.

We next look at Momentum gradient descent, shown in Figs. 5-C and 6-C. In Fig. 5-C we see a pattern in the progression of the number of intersections: every time it reaches a low value, it starts increasing. Thus, every time it finds a decent value for the number of intersections, it starts moving back toward worse ones. In the end, we end up at a fairly mediocre value. The minimum angle behaves in a similar way, every time it hits a high minimum angle, it starts decreasing. The low values of minimum angle tend to correspond to high values of the number of intersections, which, again, suggest that the two are correlated. To get a good result from this method, we have to hope that it terminates at a good solution, right before it jumps to a bad one.

Last, we look at Nesterov gradient descent, shown in Fig.s 5-D and 6-D. In this method we see a pattern similar to Momentum gradient descent. In Fig.s 5-D, we see a trend where the number of intersections decreases for a while and then jumps very high and repeats. The minimum angle follows an inverse pattern where it becomes high and then drops very low before increasing again. The jumps tend to match up across the two plots, further suggesting that the two variables are correlated. To get a good result from Nesterov gradient descent, we again have to get lucky and end on a high point in the angle which corresponds to a lower number of intersections.

In the end, we chose Vanilla gradient descent because, while it did explore bad solutions initially, eventually it found a region of good solutions. Once in this region, the solution got progressively better and in the end vanilla gradient descent returned a good solution.

Anteater makes it easy to collect the variables necessary to compare the different gradient-descent methods without any manual instrumentation. Using a traditional debugger, detecting the patterns in Momentum and Nesterov gradient descent becomes difficult; we must recall values from step to step. It may have been possible to identify these patterns by looking at the printed values alone; however, it requires significantly more mental effort from the user to try and visualized the pattern in their head. Manually generating plots similar to those presented by Anteater also requires significant effort from the user. Thus, Anteater provides a much simpler way to inspect these values.

Inspecting a Single Gradient Descent

Once we determined that Vanilla gradient descent was producing the best solution, we decided to inspect multiple runs of Vanilla gradient descent to assess its stability. We found that in most cases, the gradient descent returns a good solution. However, instances occur as shown in Fig. 7-B, where the gradient descent initially starts moving towards a bad solutions, and never recovers. In contrast, we found instances as shown in Fig. 7-A, where it immediately begins moving toward a good solution and never turns back. The instance described in the previous section falls somewhere in between these two extremes. Therefore we can conclude that although it is not the most stable, the majority of the time it produces a good solution.

Figure 7: Using Anteater to compare two runs of Vanilla gradient descent that should maximize the minimum crossing angle while minimizing edge crossings. In (A) we can see from generalized context trees, that the number of intersections rapidly decreases (the color changes from dark purple to white) while the minimum angle increases. The scatterplot shows that the descent spends its first few steps at a bad solution, takes approximately three big steps before converging on a good solution. In contrast, in (B) we see that the number of intersections is increasing throughout the descent while the minimum angle decreases. The scatterplot tells us that in general, when the number of intersections grows, the minimum angle shrinks and lands at a bad solution.

6.3 Applying Anteater to the Tennessee Eastman process

To demonstrate Anteater towards an industrial-scale application, we chose the Tennessee Eastman (TE) challenge process. Proposing an open benchmark representing a chemical plant with real-world complexity, Down and Vogel [14] presented the TE problem to encourage new advances in process control and optimization in the chemical processing industry. Since then, the TE problem has become a staple in the process-control community providing a research challenge problem for new methods and techniques to be tested against. Interested readers should look to an archive maintained by Prof. N. Lawrence Ricker for useful simulation and optimization implementations.2

By leveraging the TE challenge, we show how Anteater can help users understand a complex multi-loop model that incorporates a significant number of process manipulated variables (12) and process measurements (41). The system is set-up with two components: plant and controller. Changes in the process controller affect the performance of the plant over time. In the case study, we explored how changes in the controller set-points relate to the value function presented in Downs and Vogel [14]. We manipulated five variables in the controller, including the reactor level, the reactor temperature, the product separator level, the stripper level, and the expected product E yield. We use Anteater to explore how the variable values change and influence the cost and production of the plant.

In our optimization setup, we are searching for a parameter set that minimizes production cost while maximizing the yield of three products, denoted as F, G, and H in the TE problem. We deploy a genetic optimization algorithm to search the parameter space and use Anteater to track the process. Fig. 8 shows a few insights. Note that NaN values represent settings where the plant explodes, i.e., when the reactor pressure exceeds 3000 kPa. First, as shown in Fig. 8-A, production cost and product yields are generally correlated meaning it is feasible to identify settings with low cost and high throughput. By selecting the last iteration of the optimization in the generalized context tree, we can highlight values to which the optimization converged in the scatterplot, as shown by the black points in Fig. 8-A. The routine identifies a parameter setting with low cost and high production.

In Fig. 8-C, we see that the production of products G and H seem to be in opposition; that is high production in G causes low production in H. This makes convergence more elusive since the routine cannot satisfy both objectives simultaneously. Ideally, the algorithm would find a parameter setting that corresponds to a value that makes each of them as large as possible, but instead it tends to optimize G more heavily because its highest production is larger than that of product H.

In Fig. 8-B, we plot the parameter “Reactor temperature” against the sum of the production of all three products. We can see here that not only does a higher temperature correspond to a higher production, but the plant seems less susceptible to explosions (pressures greater than 3000 kPa) when the reactor temperature is higher. The final solution chooses a set temperature in the controller that is on the higher end.

Lastly, in Fig. 8-D, we investigate the relationships between the reactor level and the yield of product F. We can see that higher reactor levels correspond to lower production of F.

Using Anteater, we were able to gain insight into the TE simulation quickly and easily. Consider if we had tried to do the same without Anteater. In a debugger, it would have not been possible to uncover such relationships by looking at value pairs at each time step. Printing out all of the values to uncover these relationships would also fail since the sheer amount of text to read would be too large to comprehend and use to draw meaningful conclusions. If the programmer instruments the source code to write values out to files and plot them through a custom script, they could possibly draw similar insights. However, significant overhead would be required as it is unlikely that programmers would instrument the code perfectly the first time. Anteater provides a means for avoiding such overhead.

Figure 8: Anteater plots from the Tennessee Eastman case study. (A) shows that the cost and overall production are correlated and is thus possible to find a solution with low cost and high yield. The black points are the solutions reached in the final iteration of the optimization. In (B), higher values in the reactor temperature parameter correspond to better production as well as fewer plant explosions. (C) shows the production of H and G are competing against one another in the optimization, as a high G value corresponds to a low H value. In (D), the reactor level parameter influences the production of product F where lower levels in general correspond to higher production. This plot also shows that fewer explosions happen at higher reactor levels.

7 Discussion and Limitations

Omnicode vs. Anteater While Omnicode and Anteater both intend to help programmers debug and understand their programs, the two systems differ in their target audience. Omnicode aims to help novice users create mental-models to reason about their program’s execution and debug unexpected behavior. The size and complexity of programs it needs to support for this audience is quite small. Thus, Omnicode only supports programs of around 10 variables and 100 execution steps. Anteater aims to help programmers in general. Therefore it needs to support different types of programs.

While Anteater cannot support large software-systems because they produce an unmanageable amount of data, it can support much larger programs than those written by novices, such as those programs written by data scientists. Most of the differences between Omnicode and Anteater stem from the fact that they are geared toward different audiences. Omnicode supports a live programming environment because its target programs are small whereas a static environment makes more sense for Anteater. Similarly, Omnicode tracks every variable in the program whereas, for the size of program Anteater supports, this is not feasible.

Evaluation We chose to provide case studies rather than a formal evaluation because determining how to evaluate Anteater fairly has proven difficult. As discussed previously, the most directly related tool is Omnicode. While the two tools directly relate to each other, their target audiences and programs differ to the point where a direct comparison between them would not be fair to either system. Any program that is an appropriate size for Omnicode will be simple enough that using Anteater likely would not have any clear advantage over Omnicode. Any larger program appropriate for Anteater would be too large for Omnicode to support. Thus, it does not make sense to run a study comparing the two tools.

Another option for formal evaluation would have been to compare Anteater against a Python debugger, such as Pycharm. We would need to find users who have debugging experience and have them debug programs using each tool. However, if they already have debugging experience they are likely going to have an inherent bias towards the traditional debugger that they are accustomed to using. We would automatically be at a disadvantage because they would have to put in more effort to learn how to use Anteater than any standard debugger.

We could have instead used novice programmers but this would lead to a similar problem to comparing with Omnicode. Any program simple enough for a novice programmer to understand and debug would likely not lead to compelling results about the utility of Anteater. It can certainly be used on simple programs but any bug we induce on these programs for a user to find would be fairly trivial and not an accurate representation of what Anteater can do.

For future work, we can explore what is required for a fair and meaningful evaluation of this system. The fact that Anteater is not a full-scale software-package presents an additional challenge to creating a user evaluation. Running a user evaluation would require looking at a variety of evaluation patterns [15] to find one that focuses on the efficacy of the core visualization concepts of Anteater rather than its efficiency and usability as a full-scale software package.

Limitations Anteater does not scale to very large programs. While Anteater scales to larger programs than Omnicode, it still cannot handle very large, call-intensive programs. In these programs, the traces become too large and the visualizations unreadable. Research exists on collecting the entire trace of large programs [31]; future work is needed to evaluate if Anteater works well with this method.

Similarly, while Anteater can track several variables, the visualizations show at most two of them at a time. Multivariable visualization is by itself a hard research problem, and its intersection with program understanding needs further work.

Anteater assumes a sequential programming-model and currently does not extend to parallel programs, the programs it supports. Work exists in automatic tracing of parallel programs in the traditional sense (without values) but applying and extending these traces to our system is left for future work.

8 Conclusion and Future Work

We presented our prototype visualization system, Anteater, as a solution for improving the debugging and understanding of programs. By applying Anteater to three different case studies, we demonstrated how its underlying concepts present new affordances to programmers. Those affordances include 1) efficient visualization overviews of program traces, 2) new avenues for more pensive sharing, 3) the exchange of complex data programs, 4) a deeper understanding of how complex code operates. Here, we share future research directions for Anteater.

This work does not explore novel visualizations and interactions for visualizing execution traces nor does it claim to make debugging faster. Anteater demonstrates that creating a debugging system from the ground up using the principles of visualization improves the way users debug and understand programs. Now that we know this type of system is useful, in future work, we can explore creating new encodings and interactions to better visualize these execution traces as well as formally assess Anteater’s impact on debugging time. We believe that taking a visualization first perspective when creating this system opens up many possibilities for visualization and debugging.

Disclaimer

This work represents an official contribution of NIST and hence is not subject to copyright in the US. Identification of commercial systems in this paper are for demonstration purposes only and does not imply recommendation or endorsement by NIST.

Acknowledgments This work is partially supported by the NIST Graduate Student Measurement and Engineering Fellowship, through a grant with the NPSC (Award #70NANB16H141), and NSF awards IIS-1815238. We thank Tim Zimmerman, Chee Yee Tang, and Rick Candell from NIST for their helpful introduction to the Tennessee Eastman challenge problem.

Footnotes

  1. Eclipse is a popular, open IDE. See: https://www.eclipse.org/
  2. N. Lawrence Ricker, Tennessee Eastman Challenge Archive,
    https://depts.washington.edu/control/LARRY/TE/download.html

References

  1. http://clang.llvm.org.
  2. Gradient descent implementation in python returns nan. https://stackoverflow.com/questions/15211715/gradient-descent-implementation-in-python-returns-nan, 2013.
  3. E. E. Aftandilian, S. Kelley, C. Gramazio, N. Ricci, S. L. Su, and S. Z. Guyer. Heapviz: Interactive heap visualization for program understanding and debugging. In Proceedings of the 5th International Symposium on Software Visualization, SOFTVIS, pp. 53–62. ACM, New York, NY, USA, 2010. doi: 10 . 1145/1879211 . 1879222
  4. B. Alsallakh, P. Bodesinsky, A. Gruber, and S. Miksch. Visual tracing for the eclipse java debugger. In 2012 16th European Conference on Software Maintenance and Reengineering, pp. 545–548. IEEE, 2012.
  5. F. Beck, F. Hollerich, S. Diehl, and D. Weiskopf. Visual monitoring of numeric variables embedded in source code. In 2013 First IEEE Working Conference on Software Visualization (VISSOFT), pp. 1–4. IEEE, 2013.
  6. J. Bertin, W. J. Berg, and H. Wainer. Semiology of graphics: diagrams, networks, maps, vol. 1. University of Wisconsin press Madison, 1983.
  7. C.-P. Bezemer, J. Pouwelse, and B. Gregg. Understanding software performance regressions using differential flame graphs. In IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering, SANER, pp. 535–539, mar, 2015. doi: 10 . 1109/SANER . 2015 . 7081872
  8. D. Boehme, T. Gamblin, D. Beckingsale, P.-T. Bremer, A. Gimenez, M. LeGendre, O. Pearce, and M. Schulz. Caliper: Performance introspection for hpc software stacks. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’16, pp. 47:1–47:11. IEEE Press, Piscataway, NJ, USA, 2016.
  9. B. Borkholder. Mirur, 2014.
  10. M. S. T. Carpendale. Considering visual variables as a basis for information visualisation. 2003.
  11. Y.-P. Cheng, C.-Y. Ku, W.-C. Pan, C. Yang, and T.-S. Lin. Toward arbitrary mapping for debugging visualizations. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, pp. 605–608. ACM, New York, NY, USA, 2016. doi: 10 . 1145/2889160 . 2889167
  12. A. Coleman. How to quickly and effectively read other people’s code. ”https://selftaughtcoders.com/how-to-quickly-and-effectively-read-other-peoples-code/”.
  13. L. N. Q. Do, S. Krüger, P. Hill, K. Ali, and E. Bodden. Visuflow: A debugging environment for static analyses. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE ’18, pp. 89–92. ACM, New York, NY, USA, 2018. doi: 10 . 1145/3183440 . 3183470
  14. J. J. Downs and E. F. Vogel. A plant-wide industrial process control problem. Computers & chemical engineering, 17(3):245–255, 1993.
  15. N. Elmqvist and J. S. Yi. Patterns for visualization evaluation. Information Visualization, 14(3):250–269, 2015.
  16. P. Gestwicki and B. Jayaraman. Methodology and architecture of jive. In Proceedings of the 2005 ACM Symposium on Software Visualization, SoftVis ’05, pp. 95–104. ACM, New York, NY, USA, 2005. doi: 10 . 1145/1056018 . 1056032
  17. P. Gralka, C. Schulz, G. Reina, D. Weiskopf, and T. Ertl. Visual exploration of memory traces and call stacks. In 2017 IEEE Working Conference on Software Visualization (VISSOFT), pp. 54–63. IEEE, 2017.
  18. A. Hidayat. http://esprima.org.
  19. J. Hoffswell, A. Satyanarayan, and J. Heer. Augmenting code with in situ visualizations to aid program understanding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 532. ACM, 2018.
  20. S. Jarman. Tips for navigating large and unfamiliar codebases. ”https://hackernoon.com/tips-for-navigating-large-and-unfamiliar-codebases-f5426cea552c”, 2017.
  21. JetBrains. Pycharm, the python ide for professional developers, 2000.
  22. M. Kahng, N. Thorat, D. H. P. Chau, F. B. Viégas, and M. Wattenberg. Gan lab: Understanding complex deep generative models using interactive visual experimentation. IEEE transactions on visualization and computer graphics, 25(1):310–320, 2019.
  23. H. Kang and P. J. Guo. Omnicode: A novice-oriented live programming environment with always-on run-time value visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 737–745. ACM, 2017.
  24. B. Karran, J. Trümper, and J. Döllner. Synctrace: Visual thread-interplay analysis. In Proceedings of the 1st Working Conference on Software Visualization, VISSOFT, p. 10. IEEE Computer Society, 2013. doi: 10 . 1109/VISSOFT . 2013 . 6650534
  25. H. Lam, M. Tory, and T. Munzner. Bridging from goals to tasks with design study analysis reports. IEEE transactions on visualization and computer graphics, 24(1):435–445, 2018.
  26. T. D. LaToza and B. A. Myers. Developers ask reachability questions. In Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering-Volume 1, pp. 185–194. ACM, 2010.
  27. H. Lieberman and C. Fry. Bridging the gulf between code and behavior in programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’95, pp. 480–486. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 1995. doi: 10 . 1145/223904 . 223969
  28. S. Litvinov, M. Mingazov, V. Myachikov, V. Ivanov, Y. Palamarchuk, P. Sozonov, and G. Succi. A tool for visualizing the execution of programs and stack traces especially suited for novice programmers. arXiv preprint arXiv:1711.11377, 2017.
  29. M. Meyer, T. Munzner, and H. Pfister. Mizbee: a multiscale synteny browser. IEEE transactions on visualization and computer graphics, 15(6):897–904, 2009.
  30. T. Ohmann, R. Stanley, I. Beschastnikh, and Y. Brun. Visually reasoning about system and resource behavior. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, pp. 601–604. ACM, New York, NY, USA, 2016. doi: 10 . 1145/2889160 . 2889166
  31. G. Pothier, E. Tanter, and J. Piquer. Scalable omniscient debugging. In Proceedings of the 22Nd Annual ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications, OOPSLA ’07, pp. 535–552. ACM, New York, NY, USA, 2007. doi: 10 . 1145/1297027 . 1297067
  32. S. P. Reiss. The challenge of helping the programmer during debugging. In 2014 Second IEEE Working Conference on Software Visualization, pp. 112–116, Sep. 2014. doi: 10 . 1109/VISSOFT . 2014 . 27
  33. M. Renieris and S. P. Reiss. ALMOST: Exploring program traces. In 1999 Workshop on New Paradigms in Information Visualization and Manipulation, pp. 70–77. ACM, 1999. doi: 10 . 1145/331770 . 331788
  34. D. Rozenberg and I. Beschastnikh. Templated visualization of object state with vebugger. In 2014 Second IEEE Working Conference on Software Visualization, pp. 107–111. IEEE, 2014.
  35. R. Schulz, F. Beck, J. W. C. Felipez, and A. Bergel. Visually exploring object mutation. In 2016 IEEE Working Conference on Software Visualization (VISSOFT), pp. 21–25, Oct 2016. doi: 10 . 1109/VISSOFT . 2016 . 21
  36. B. Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In The Craft of Information Visualization, pp. 364–371. Elsevier, 2003.
  37. D. Socha, M. L. Bailey, and D. Notkin. Voyeur: Graphical views of parallel programs. In Proceedings of the 1988 ACM SIGPLAN and SIGOPS Workshop on Parallel and Distributed Debugging, PADD, pp. 206–215. ACM, New York, NY, USA, 1988. doi: 10 . 1145/68210 . 69235
  38. Sourcegraph. Sourcegraph: Search, navigate, and review code, 2018.
  39. J. Sundararaman and G. Back. Hdpv: Interactive, faithful, in-vivo runtime state visualization for c/c++ and java. In Proceedings of the 4th ACM Symposium on Software Visualization, SoftVis ’08, pp. 47–56. ACM, New York, NY, USA, 2008. doi: 10 . 1145/1409720 . 1409729
  40. R. Tiarks and T. Roehm. Challenges in program comprehension. Softwaretechnik-Trends, 32(2):19–20, 2012.
  41. J. Trümper, J. Bohnet, and J. Döllner. Understanding complex multithreaded software systems by using trace visualization. In Proceedings of the 5th International Symposium on Software Visualization, SOFTVIS, pp. 133–142. ACM, New York, NY, USA, 2010. doi: 10 . 1145/1879211 . 1879232
  42. C. Williamson and B. Shneiderman. The dynamic homefinder: Evaluating dynamic queries in a real-estate information exploration system. In Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 338–346. ACM, 1992.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
407152
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description