Output-Directed SVG Programming

Output-Directed SVG Programming

Brian Hempel    Justin Lubin    Ravi Chugh

Output-directed programming is a paradigm whereby users directly manipulate the output they wish to create while the system automatically builds a text-based program that produces the output. We present new output-directed programming techniques in a system called Sketch-n-Sketch, in which programs generate Scalable Vector Graphic (SVG) designs. We show how these interactions enable a variety of complex, readable programs to be constructed with a high degree of automation, often without requiring any text-based program editing. Compared to prior efforts in this direction, our techniques are applicable at more stages of program construction, facilitating a greater diversity of programs which may be created entirely by direct manipulation. We evaluate our techniques by implementing over a dozen parametric designs, including several from the test suite proposed in Watch What I Do: Programming by Demonstration. While some of our output-directed programming interactions are by necessity domain-specific, others are implemented in a generalized way, to support their application to other output-directed programming domains. This work is a milestone in the expressiveness of output-directed programming and, thus, contributes to ongoing efforts to blend programming languages and direct manipulation interfaces.

Output-Directed Program Synthesis, Direct Manipulation, Sketch-n-Sketch

Anonymous University of Chicago, USAbrianhempel@uchicago.edu University of Chicago, USAjustinlubin@uchicago.edu University of Chicago, USArchugh@cs.uchicago.edu \ccsdesc[300]Software and its engineering Integrated and visual development environments \ccsdesc[300]Human-centered computing Human computer interaction (HCI) \category\relatedversion\supplement\funding\EventEditorsJohn Q. Open and Joan R. Access \EventNoEds2 \EventLongTitle42nd Conference on Very Important Topics (CVIT 2016) \EventShortTitleJanuary 2019 \EventAcronymECOOP \EventYear2019 \EventDateJanuary, XX–XX, 2019 \EventLocationLittle Whinging, United Kingdom \EventLogo \SeriesVolume42 \ArticleNo000

1 Introduction

Text-based programming presents a steep learning curve to novices as they approach programming, and it hobbles experts even as they build complex software systems.

In response, many techniques have been developed to facilitate programming more “directly” than by writing text—such as programming-by-example (PBE), programming-by-demonstration (PBD), and constraint-oriented programming (COP). These techniques have succeeded in lowering the barriers of programming for creating complex artifacts in a variety of domains, but—as discussed in § 5—they generally limit which building blocks are available to programmers. Ideally, programming environments could retain the power and full flexibility of text-based programming while providing new capabilities for directly turning human intent into code.

Sketch-n-Sketch is a prior programming system for creating Scalable Vector Graphics (SVG) drawings that attempts to achieve this combination by augmenting an ordinary text-editing environment with the following additional interactive mechanisms for transforming code: (a) “semi-automated programming” techniques [17], which provide the ability to directly create and manipulate output values—these user interactions are co-designed with corresponding program transformations that make “large” changes to codify the user’s actions; (b) bidirectional programming techniques [8, 28], which allow “small” program changes to be specified by small changes to the program’s output; and (c) structured editing techniques [18], which add graphical, direct manipulation actions (namely, mouse-based multi-selection of expressions for invoking refactorings) to the source code text editor itself. These techniques aim to form an array of interactive program editing capabilities for a variety of programming tasks, application domains, and user scenarios.

Prior Output-Directed Programming Techniques in Sketch-n-Sketch

The subject of this paper is to develop further the first of these interactive mechanisms—the semi-automated programming category of techniques established by Hempel and Chugh [17]—which we dub output-directed programming. With their approach to output-directed programming, Sketch-n-Sketch provides direct manipulation tools for drawing new shapes (by adding new definitions to the program), relating the attributes of shapes (by introducing new shared variables and arithmetic expressions), and grouping and abstracting shapes (by transforming individual definitions into reusable functions parameterized over design choices). These “Draw-Relate-and-Abstract” output-directed programming tools are used to construct several high-level, readable programs to generate non-trivial designs.

Although a useful step towards the goals of output-directed programming for the domain of SVG, the prior techniques in Sketch-n-Sketch [17] exhibit significant drawbacks both in approach and in implementation. Regarding the approach, only final output values are visualized and subject to direct manipulation. As programs grow, however, output values constitute only a small fraction of program execution that may be of interest to the user. Furthermore, the user cannot specify what areas of the potentially large program should (or should not) change based on the output actions.

Regarding implementation, the set of output-directed Draw-Relate-and-Abstract tools in Sketch-n-Sketch is small, limiting the kinds of SVG designs that benefit from output-directed program construction. Utility of the tools is further limited by the fact that many of the program transformations operate only on a restricted syntactic subset of the (albeit general-purpose) language—namely, a series of top-level definitions followed by a list literal “main” expression that refers to the top-level definitions. If the program veers out of this language subset, those tools become unavailable. As a result of these drawbacks, even within the domain of SVG drawings, the output-directed capabilities of Sketch-n-Sketch are restricted to a small set of programming tasks.

Our Approach: New and Improved Output-Directed Programming Techniques

To further the vision established by Hempel and Chugh [17], we design and implement a new approach to output-directed programming in Sketch-n-Sketch to address the aforementioned limitations.

In our approach, we add two new kinds of user interface elements (beyond the final output of programs) for users to interact with. First, we add intermediate execution products to facilitate manipulation of partial programs (with yet-unspecified output) and large programs (where many relevant intermediate values do not appear directly in the output). Second, because there are usually many reasonable program changes given a user action, we allow the user to identify a focus expression to delineate the syntactic scope of desired changes to the program. Focused editing reduces the degree of ambiguity for several transformations (e.g., whether to add a new definition at the top-level or within an existing definition) and also enables the construction of recursive functions.

Our implementation significantly extends Sketch-n-Sketch’s set of output-directed tools, some of which are SVG-specific program transformations but many of which are general-purpose refactorings. Furthermore, to relax the syntactic constraints of the prior work, we employ several run-time tracing techniques that record value provenance, used by tool implementations to provide meaningful program changes despite the variety of programming styles afforded by the general-purpose source language.


To summarize, this paper makes the following contributions:

  1. We propose that output-directed programming systems should allow users to interact with—and co-designed program transformations to consider—intermediate execution products and focused expressions in addition to the final program output.

  2. We extend the Sketch-n-Sketch system [17] with a large set of new tools which embrace the new user interface elements above. These tools also employ more flexible provenance techniques in order to operate on a greater variety of program structures and, thus, may later be more readily adapted to non-visual output domains. To our knowledge, our system is the first to offer (a) output-directed tools for common, general refactorings, and (b) recursion by demonstration in a setting where the underlying representation of the artifact is a general-purpose programming language.

  3. We evaluate the new Sketch-n-Sketch system with 16 SVG design tasks, drawn from Watch What I Do: Programming by Demonstration [1] and other sources. Our system can construct reusable programs for these tasks entirely through direct manipulation. These programs (available in Appendix B) span more than 400 lines of code in total.

Our contributions advance the frontier of what expert users (i.e., the authors) can achieve solely through output-directed programming, thus contributing to ongoing efforts to blend programming languages and direct manipulation interfaces. Further refinement of our techniques to allow other experienced programmers to, for example, “walk up and use” the system [32] is left to future work.

Paper Outline

Our user interface and program transformation techniques are best described by example. We start by walking through a simple example in its entirety (§ 2). After this overview, we describe additional direct manipulation tools in Sketch-n-Sketch, as well as implementation techniques to facilitate co-designed program transformations (§ 3). Then, we describe several additional notable interactions and transformations among our benchmark examples, as well as limitations of the system (§ 4). Following a discussion of related work (§ 5), we wrap up with a discussion of limitations and avenues for future work on output-directed programming, both for the domain of SVG as well as other more general-purpose settings (§ 6).

In the rest of the paper, we refer to our new system simply as Sketch-n-Sketch, without qualification, except when comparing to the prior version.

2 Overview

Figure 1: A recursive program to generate a Koch snowflake fractal, built solely by output-directed program transformations in Sketch-n-Sketch (without any text-based programming).

Sketch-n-Sketch is a bimodal programming environment, as depicted in Figure 1: the left pane is an ordinary source code text editor; the right pane renders the SVG design generated by the program on the left and also offers a graphical user interface for performing transformations on the SVG output (i.e., with mouse-based manipulations). The programmer may also perform keyboard text-edits on the code at any time during program construction. Because the focus of our work is output-directed programming, we will describe an example program that is created almost exclusively by output-directed transformations; we will resort to text-editing only twice, both times because of incidental limitations of Sketch-n-Sketch’s color picker. In § 4, we demonstrate programs created solely via output-direct manipulations.

Figure 2:
Lambda logo.

To introduce how the Sketch-n-Sketch workflow enables one to build a program through output-directed manipulations, we will walk through the construction of a reusable design for the Sketch-n-Sketch logo (Figure 2). This design is a superset of an example from the prior version of Sketch-n-Sketch [17, Overview]; to keep the presentation here self-contained, we defer an explicit comparison until § 4. We will build the logo design in four consecutive phases: (i) drawing the shapes, (ii) relating graphical features to align shapes and equate colors, (iii) abstracting the design into a reusable function, and, finally, (iv) refactoring the design to clean up the generated code. Throughout, the code shown is as produced by Sketch-n-Sketch. Newlines inserted for formatting purposes in this paper are “escaped” by a “\” backslash. The final code for this example is shown in Figure 12. The Supplementary Materials include a URL to an unnarrated screencast of this example.

2.1 Drawing Shapes

Figure 3:
Initial toolbox.

The initial program template provided by Sketch-n-Sketch is nearly blank, defining only an empty list of SVG shapes:

svg (concat [

Starting from this bare program, the programmer would first like to get some shapes into the program’s output. A traditional programming environment would require looking up the drawing library’s documentation and perhaps copy-pasting some example code into the program.

The Sketch-n-Sketch output pane, however, imitates a traditional drawing application. The programmer clicks on the “square” tool from the toolbox (Figure 3) and drags the mouse on the canvas. When she releases the mouse, a new square1 definition is inserted in the program representing a square that is sized and positioned appropriately. A square1 variable usage is also inserted into the shape list at the end of the program, and thereby a red square appears in the program output on the canvas (not shown).

square1 = square 0 [158, 127] 156
svg (concat [

All new shapes are by default drawn in red (the color number 0 in the code above). The programmer performs a text-edit to change the square color to "#fcc712" for a desired shade of gold, and presses the “Run” button to re-run the program with the new color.

Figure 4:
Snap-drawing two lines to features of a rectangle.

To create the lines for the lambda symbol, the programmer “snap-draws” [15] between features of the gold square. With the “line” tool selected, she draws one line from the top-left to the bottom-right corner and a second line from the bottom-left corner to the center point (shown in the top and bottom, respectively, of Figure 4). Sketch-n-Sketch interprets these snaps as constraints which should always hold, i.e., the lines should still coincide with the corners and center even if the square is moved or resized. Sketch-n-Sketch encodes these constraints in the program via shared variables: three new variables are introduced for the square’s top-left and coordinates and its width , and the spatial properties of both the square and the two lines are defined in terms of simple math on those variables (shown in Figure 5).

Note that Sketch-n-Sketch’s name generator usually chooses reasonable names, but here it has called the coordinate num; the programmer will remedy this minor defect later. Note also how the new variable definitions are positioned so their scopes “tightly wrap” the variable uses. If the programmer wished to rearrange the definitions, she could use either ordinary text-editing or the structured editing features provided by Sketch-n-Sketch [18].

y = 127
num = 158
topLeft = [num, y]
w = 156
square1 = square  "#fcc712" topLeft w
y2 = y + w
line1 = line 0 5 topLeft [ num+ w, y2]
line2 = line 0 5 [num, y2] [ (2! * num + w)/ 2!, (2! * y + w) / 2!]       ...
Figure 5: Code after drawing one square and snap-drawing two lines. Freeze annotations, written !, instruct Sketch-n-Sketch not to change those constants when shapes are moved on the canvas [8].

2.2 Relating Properties

Figure 6: Clicking sliders to select colors.

The programmer would like to require that the color and width of the two lines are always identical. Sketch-n-Sketch offers tools for relating features after they are drawn. To relate the colors, the programmer first selects the two lines (indicated by yellow highlights in Figure 6) to expose sliders for each line’s color and stroke width. When selected, notice how Sketch-n-Sketch displays the names of the lines (line1 and line2) adjacent to the corresponding output shapes on the canvas. She can manipulate the sliders to change associated constants in the program, but here she instead clicks the whole line2 color slider to select the property itself (selected properties are indicated in green, as visible in the Figure 6) and then selects the other line’s color slider as well. (Dragging and clicking-to-select sliders are features inherited from prior versions of Sketch-n-Sketch [8, 17]).

Figure 7: Equalizing selected colors.

Whenever a selection is made upon the canvas, Sketch-n-Sketch displays a floating menu of output tools (shown in Figure 7). The programmer would like the two selected colors to always be the same, so she hovers the Make Equal tool with her mouse, revealing a list of possible program transformations in a submenu—in this case there is only one, but if the lines had been different colors two results would be offered to let the programmer select which line’s color should take priority. Hovering the “Equalize by removing line2 color” result in the submenu previews a diff of the change in the code box (not shown)—in this case the Make Equal tool will introduce an appropriately-named color variable used for both lines. The programmer is happy with this change and clicks to apply it. She then repeats this workflow to equalize the stroke width of the two lines as well.

2.3 Abstracting the Design

The programmer would like to make her logo design reusable—in other words, she would like to create a function that, given size and color arguments, generates a logo appropriately.

She gathers the three shapes into a single expression by selecting the shapes and invoking the Group tool in the floating menu. Analogous to the grouping functionality of traditional graphics editors, Group in Sketch-n-Sketch gathers the shapes into a single list definition, which is used in place of the individual shapes in the final shape list; notice, again, how Sketch-n-Sketch attempts to choose a new variable name that is informative:

squareLineLine = [square1, line1, line2]
svg (concat [
Figure 8: List widgets.

Now that the shapes are grouped together into a list, Sketch-n-Sketch offers that list itself as an object for selection on the canvas. Lists are represented on the canvas with list widgets—boxes with dotted borders (intended to evoke the elements of a list). When the programmer hovers her mouse over a shape, list widgets are shown for each of the list expressions in the program in which the shape appeared. In this example there are four such lists: the initial list literal [square1, line1, line2] plus three expressions at the end of the program that evaluate to lists (the squareLineLine variable usage, the list literal [squareLineLine], and the flattened list value produced by the functional call concat [squareLineLine]). Notice how Sketch-n-Sketch chooses a layout to avoid otherwise-overlapping widgets. Hovering over a list widget highlights the corresponding code expression in the program (code highlights are not shown here), allowing the programmer to distinguish the various lists. As shown in Figure 8, she selects the list widget corresponding to the [square1, line1, line2] literal where the squareLineLine binding is introduced.

squareLineLineFunc y num w color strokeWidth =
  let topLeft = [num, y] in
  let square1 = square  "#fcc712" topLeft w in
  let y2 = y + w in
  let line1 = line color strokeWidth topLeft [ num+ w, y2] in
  let line2 = line color strokeWidth [num, y2] \
                [ (2! * num + w)/ 2!, (2! * y + w) / 2!] in
  [square1, line1, line2]
squareLineLine = squareLineLineFunc y num w color strokeWidth
Figure 9: Code after equalizing colors, grouping three shapes, and abstracting the group.

With that list selected, she invokes the Abstract tool to construct a function that returns the selected item, namely the list [square1, line1, line2]. The Abstract tool heuristically pulls in bindings used only in the construction of [square1, line1, line2] to produce a function named squareLineLineFunc parameterized over y, num (that is, ), w, color, and strokeWidth, as shown in Figure 9.

Figure 10:
User-defined drawing tool.

The programmer has created a reusable function, which she could manually call multiple times in the program. Conveniently, however, Sketch-n-Sketch’s type inference notices that this function accepts an -coordinate and a width, so this function appears in the tool box on the right-hand side as a custom drawing tool (Figure 10). The programmer selects the tool and draws two more copies of the logo on the canvas—Sketch-n-Sketch inserts appropriate function calls into the program automatically, with default arguments for color and strokeWidth.

squareLineLineFunc1 = squareLineLineFunc 273 422 70 0 5
squareLineLineFunc2 = squareLineLineFunc 407 135 123 0 5

Although happy that the design is reusable, the programmer needs only one copy for now. Selecting the list widgets for the two new logos and hitting the Delete key removes the corresponding calls to the newly-created function.

2.4 Refactoring

Although the programmer has produced a reusable design, she may change her mind about particular details in the program. For example, the programmer may like to clean up the code a little bit (the name num for an coordinate is particularly bothersome) and also may want to add a border to the design. Sketch-n-Sketch provides features to help implement code changes such as these two.

Similar to list widgets, call widgets—drawn with solid borders—are placed around items that are produced by function calls in the program. The programmer clicks the call widget for the first call to squareLineLineFunc, which (a) exposes the function’s arguments above the called widget, and (b) focuses any changes to just that function. The rest of the program output (there is none currently) disappears from the canvas for the duration of the focused editing session.

Figure 11: Rename focused function.

With the function focused, the programmer clicks the function’s name label exposing a text box which allows her to rename the function (Figure 11). She inputs the name logoFunc and hits the Return key to invoke a standard Rename refactoring. The arguments can be renamed as well—she renames num to x, then also clicks the Reorder Argument arrows to place the x argument before y.

Now the programmer would like to add a border. After changing the two existing lines to black via color sliders, she then snap-draws a polygon to the corners of the square with the “Polygon” tool from the toolbox. Because she is focused on this function, the new polygon, called polygon1, is added to the return expression of the logoFunc function instead of to the program’s final shape list (see lines 20-24 of Figure 12). She text-edits the fill color of the polygon to "none" and adjusts the stroke width of the polygon using a slider.

The lambda logo program is now completed to the programmer’s satisfaction; the final code is shown in Figure 12. Except for a couple of color changes, the programmer was able to create the program entirely through output-directed manipulations on the canvas by using tools for drawing, relating, abstracting, and refactoring.

2y = 127
4num = 158
6w = 156
8color = 360
10strokeWidth = 5
12logoFunc x y w color strokeWidth =
13  let topLeft = [x, y] in
14  let square1 = square  "#fcc712" topLeft w in
15  let y2 = y + w in
16  let xYPair = [ x+ w, y2] in
17  let line1 = line color strokeWidth topLeft xYPair in
18  let line2 = line color strokeWidth [x, y2] \
19                [ (2! * x + w)/ 2!, (2! * y + w) / 2!] in
20  let polygon1 =
21    let pts = [[x, y], [ x+ w, y], xYPair, [x, y2]] in
22    let [color, strokeColor, strokeWidth] = [ "none", 360, 5] in
23      polygon color strokeColor strokeWidth pts in
24  [square1, line1, line2, polygon1]
26squareLineLine = logoFunc num y w color strokeWidth
28svg (concat [
29  squareLineLine
Figure 12: The final code for the lambda logo overview example (Figure 2), produced almost entirely by draw, relate, abstract, and refactor output-directed manipulations on the canvas.

3 Design and Implementation

Having worked through an example workflow, we now describe in more detail the design and implementation of Sketch-n-Sketch that make those interactions possible. Sketch-n-Sketch is an in-browser application written in the functional programming language Elm [12] extended with a few features and libraries. Our implementation is available in the Supplementary Materials (Appendix A).

Figure 13 depicts the workflow for output-directed programming in Sketch-n-Sketch. The programmer-facing language for Sketch-n-Sketch users is a standard functional language with Elm-like syntax. An ordinary text-based program is evaluated and its SVG output is rendered graphically on the output canvas.

Figure 13: Output-directed programming workflow.
(A) Program evaluates to value of type SVG.
(B) Graphical widgets for manipulating the output value, intermediate values, and the program.
(C) Programmer selects from toolbox (cf. Figure 3) and draws on canvas, or manipulates existing shapes and selects a tool from the floating tools menu (cf. Figure 7).
(D) One or more candidate programs are produced; the programmer previews the changes and then chooses one (cf. Figure 18).

The goal of output-directed programming is to offer “direct” interactions with the evaluation of the program to affect changes to the program source code. Like in the prior approach [17], the programmer may, conceptually, select arbitrary sub-values in the final output before invoking some program transformation; the value selections are “arguments” to the transformation and are interpreted in a transformation-specific way.

In our approach, the programmer can also select additional kinds of terms. First, the programmer may select values produced by intermediate execution steps. Second, the programmer may select sub-expressions in the program. These two kinds of selections—intermediate values and program sub-expressions—also serve as arguments to the program transformations.

In the rest of this section, we describe the SVG-specific user interface and program transformations currently implemented in Sketch-n-Sketch.

3.1 User Interface

The user interface renders three kinds of graphical, SVG-specific widgets.

Output Value Widgets

As inherited from prior versions of Sketch-n-Sketch [8, 17], we render graphical features for different kinds of SVG primitives (e.g., rect, line, circle). For example, Figure 6 depicts color widgets for fill and stroke colors. As shown in Figure 4, nine point feature widgets are drawn for rectangles; some of these (e.g., the top-left corner) correspond directly to , , , and values in the output (“primitive features”), while others (e.g., the bottom-right corner and the center) are derived from these values (“derived features”). Two distance widgets are also drawn for selecting the and values. For lines, point widgets are drawn for the endpoints (primitive features) as well as the midpoint (a derived feature). New to our implementation, distance widgets are drawn between all pairs of selected point widgets, as depicted in a subsequent example (cf. Figure 20).

Intermediate Value Widgets

In addition to widgets representing sub-values of the final output, in this work we draw widgets for several common kinds of intermediate values produced during the execution of SVG-drawing programs. We draw list widgets for any intermediate value that is a list of points or SVG values (cf. Figure 8). Additionally, we draw point widgets for all intermediate point values and offset widgets for values computed by adding a numeric value to an or coordinate (as depicted in Figure 27 for a subsequent example). Like with primitive and derived shape features of the final output, we support snapping to these helper widgets.

Expression Widgets

The selection of program expressions can be used to control the syntactic scope of changes made to the program; for example, in § 2.4, the programmer wanted to add a new shape within an existing definition rather than at the top level of the program. Currently, we provide one way of establishing an expression focus, applicable to function calls that produce point or SVG values: for any such expression in the program, we draw a call widget around its SVG value (cf. Figure 11). When clicking the border of the call widget, the function definition becomes the focus, and the arguments to the selected call are used for rendering the widget. In § 4, we will explain how focused expressions also enable recursive drawing.

Widget Labels and Visibility

To aid comprehension, labels are drawn next to most widgets on the canvas. A label shows the expression associated with a widget or, if the expression is bound to a variable (i.e., the expression is itself a variable usage or is the right hand side of a let-binding), then the label displays the variable name; in such cases the label may be clicked to rename the variable.

Because program execution may involve a large number of intermediate evaluation steps, even simple programs might clutter the canvas with widgets, making it unusable. Therefore, our current implementation hides most widgets by default and uses heuristics to determine when to show them—generally, upon the mouse hovering over some associated shape (i.e. value). Additionally, widgets from intermediate expressions in standard library code—outside the visible program—are generally not shown. In the future, these visibility choices could be more systematically controlled, e.g., by source code annotations or user interface options.

3.2 Program Transformations

Draw Shape
Draw Custom Shape (“Lambda” [17])
Draw Point
Draw Offset
Focused Drawing
Snap Drawing (Mostly UI Concern)
Make Equal
Distance Features (Mostly UI Concern)
Repeat over Function Call
Repeat over Existing List
Repeat by Indexed Merge
Fill Hole
List Widgets (Mostly UI Concern)
Rename in Output
Add/Remove/Reorder Argument
Reorder List
Add to Output
Select Termination Condition
Figure 14: User interface features and program transformations in Sketch-n-Sketch. Those marked with ★ are new to our implementation; the rest are improved versions of those in [17].

Figure 14 lists the new and improved program transformation tools in our version of Sketch-n-Sketch. The example in § 2 demonstrated several tools in action; § 4 will highlight several more. Some tools are SVG-specific, but many are implemented in a generalized way. Here, we comment on a few noteworthy aspects of our implementation.


Conceptually, each program transformation has access to the derivation tree—the program and the complete sequence of evaluation steps—as well as the selections, and then decides how to transform the program. In our implementation, however, tool implementations do not manipulate derivation trees directly. Instead, the evaluator records run-time traces to track provenance for different kinds of values. This provenance includes the pieces of information from the derivation tree that are needed by our current set of program transformations.

To support the tools summarized in Figure 14, our current approach relies on several kinds of provenance attached to values during evaluation. “Dataflow-only” traces [8] record the flattened sequence of primitive arithmetic operations that produced numeric values. For example, when the programmer invokes the Make Equal tool, the dataflow traces for the selected values are interpreted as arithmetic expressions and used to form an appropriate system of equations to be solved by the REDUCE computer algebra system [16].

Additional provenance tracing has been added to support our new tools. First, we track “based-on” provenance, which, in order to deduce which expressions a selected value is based on, tracks a (large) subset of the pairs in each value’s derivation tree, partially ordered by their locations in the tree. Notably ignored are certain control flow features such as the conditional expression of a branch and the left side of a function application; this choice is based on the intuition that selecting a shape, for example, should refer to the particular function call that produced the shape and not the shape function itself. The resulting partially ordered pairs are useful for a variety of tools. In general, tool implementations prefer dependent expressions that are tightest in scope with respect to the selected values being manipulated (i.e. expressions encountered later in execution), but retaining a large amount of information from the derivation tree gives tools the flexibility to interpret selections contextually. For example, the Add Argument tool looks at all the expressions a selection is “based on”, retains the expressions that are inside the body of the function focused for editing, and then lets the programmer disambiguate which of those expressions was intended to become a function argument by providing multiple possible transformed programs in the tool’s submenu (one for each expression).

Second, we track “parents” provenance, which can be thought of as recording the pattern matches used to deconstruct and access container values (e.g. lists). This information is used, for example, to relate the individual numeric components of a point with the point itself.

While these tracing mechanisms are sufficient to support the reasoning needed by our current tools, it will be important in future work to develop more extensible and composable forms of provenance (e.g., perhaps by extending [33, 2]) to make it easier to define new transformations in a way that interacts well with existing ones (discussed further in § 6).

Value Holes

We extend our source-level language with a mechanism called value holes that helps to implement the desired functionality of several tools. A value hole is a sequence of zero or more value branches followed by an optional default value branch as well as a boolean flag. When evaluating a value hole for the -th time, the result is the value of its -th branch. If there are fewer than branches and a default value is provided, the result is the default value; otherwise, the program crashes. The boolean flag determines how the “-th time” is computed: if the flag is false, then is the number of times the value hole has been encountered in the left-to-right, call-by-value evaluation; otherwise, is the number of times the current function appears in the call stack.

Several tools use value holes internally. Most notably, snap-drawing uses value holes to simplify inserting template code that honors the output-directed snaps. For example, if the programmer draws a shape expecting one point of the new shape to snap to an existing point on the canvas, internally Sketch-n-Sketch will insert the new shape into the program with a pair of value holes in place of the snapped point [, ]. Before showing the code to the programmer, the provenance of the default value in each hole is inspected and the hole is replaced by an expression that enforces the snap. In practice, the hole resolver will first attempt to replace a super-expression of multiple value holes such that e.g. the aforementioned pair of holes might be replaced with a single variable (e.g. point1) instead of two expressions in a pair (e.g. [x1, y1]).

These default-branch-only value holes are used for mixing constraints with template code and are never shown to the programmer. We also use value holes in two other scenarios, both of which result in value holes being inserted into programmer-visible code: first, to support recursive drawing, and, second, to support a repetition-by-example workflow. Both scenarios will be presented in § 4.


We want user-defined functions to be drawable with the mouse without requiring annotations to indicate which arguments are, e.g., the width and height. During type inference, we tag the types at code locations with additional side information explaining the expression’s role. The standard library drawing functions are already annotated with roles, so using these functions causes role information to flow back into the program. A similar scheme called brands is used in [31], although we additionally apply certain built-in usage rules—e.g. a number added to an X value must be a HorizontalDistance—to infer roles in more cases.

4 Evaluation

Figure 15: Example programs created solely via output-directed manipulations.
LOC = Source lines of code   = Final design is a function that appears as a drawing tool
Comparison to [17]: If an example can similarly be created in [17], code metrics for [17] appear in parentheses. Math operations (MO) indicate code simplifications due to improvements in this work.  ★ (resp. ✩) = Task cannot (resp. can with undesired parameterization) be created in [17]
Source of examples:
WWID: PBD [1]: Tasks marked with underline (dashed = only partially completed)
A Lillicon [4]: (ix) Battery   A Sketch-n-Sketch [17]: (xi) Logo (via Three Tris)
A QuickDraw [4]: (x) Ladder   A Sketch-n-Sketch [8]: (xii) N Boxes, (xiii) Ferris Wheel

To evaluate the new techniques in Sketch-n-Sketch, we implemented 16 parametric designs (summarized in Figure 15) drawn from the following sources:

  • Six examples (underlined in Figure 15) are from Watch What I Do: Programming by Demonstration [1]. The WWID: PBD test suite is diverse: 15 of its 32 tasks may be interpreted as parametric drawings. Of those 15, our system completes 4 and partially completes another 2.

  • We take 5 tasks from other drawing literature [4, 7, 8, 17].

  • We additionally implement 5 novel tasks.

To isolate the expressive power of Sketch-n-Sketch’s tooling, all 16 programs were built entirely via output-directed manipulations, without any text editing in the code box. In total, the final programs span 427 lines of code (as shown in Appendix B).

Below, we describe and evaluate our examples as follows. First, we compare the present work to the prior version of Sketch-n-Sketch by Hempel and Chugh [17]: we explain how the lambda logo example presented in § 2 differs from the workflow in [17, Overview], and then briefly discuss the 4 of the 16 evaluation examples that can also be completed (at least partially) in [17]. Second, we walk through key steps in three examples (Figure 15i-iii) to demonstrate features not exercised by the overview example, notably point widgets, offset widgets, repetition over points, recursive drawing, and repetition by demonstration. Finally, we describe limitations of our current system by discussing tasks from [17] and the WWID: PBD benchmark suite that our current system cannot complete without text edits.

4.1 Comparison to Prior Work

Compared to the previous version of Sketch-n-Sketch upon which we build [17], we add several new features, summarized in Figure 14. Along with new features—exposing certain intermediate execution products for manipulation, the ability to focus expressions, and a suite of output-directed repetition and refactoring tools—this work also modifies and improves upon the Draw-Relate-and-Abstract tooling present in [17].

We discuss these improvements below, first by comparing the lambda logo example workflow (see Appendix A for videos), and then by briefly considering the four examples in Figure 15 that can also be constructed in [17] using only output-directed manipulations.

Overview Example Differences

Minus the text-edit for a specific shade of gold and the addition of a border, the lambda logo from the overview example is the same as in [17, Overview]; the prior Sketch-n-Sketch system completes the design without text edits. We demonstrated the same example primarily because it is a good introduction to the Sketch-n-Sketch workflow. Nevertheless, there are a few differences throughout the draw, relate, abstract, and refactor steps. Below, we discuss differences relevant to this example.


In the present work, the drawing tools now support snap-drawing to existing points during a drawing operation. In [17], snapping had to be affected after-the-fact with the Make Equal tool. Additionally, the ability to delete shapes from the output is also new—previously one was required to delete the code manually.


The previous Sketch-n-Sketch [17] did not offer previews of transformations before applying them, nor did it offer multiple possible transformation results. Letting the programmer choose among multiple results is occasionally necessary when there is more than one way to transform a program to enforce a constraint.


Like the prior Sketch-n-Sketch [17], the programmer can turn a design into a function by invoking Group and then Abstract. Looking forward to applying output-directed programming to domains beyond vector graphics, in this work we have focused on reimagining the tools so they operate on primitive program structures (lists, let-bindings, functions) rather than on the domain-specific syntactic constructs required by [17] (top-level bindings only, each with a strict left-top-right-bot parameterization). Consequently, Group now simply puts the selected items into a list, whereas in prior work Group performed a complicated rewrite to honor the required left-top-right-bot parameterization. Similarly, the Abstract tool in this work simply attempts to build a function around the relevant expression, guided by its free variables, whereas prior work assumed the function body was already fully present as some top-level definition and abstracted that top-level definition over its named numeric constants.

Moreover, the prior work was unable to infer default arguments for the custom drawing functions, instead requiring that an existing call to the function was already present from which to copy default arguments. If the function was in the code but not called, it could not be used as a custom drawing tool. In this work, we automatically infer roles (§ 3.2) to choose reasonable default arguments.


All output-directed refactoring tools (Figure 14), such as Rename and Reorder Argument, are new. Focused editing is novel as well, as are facilities for adding or removing shapes to or from a function’s return expression.

Evaluation Example Differences

Of the 16 evaluation examples in Figure 15, four of the examples (iv, v, vi, ix) can be constructed in the prior Sketch-n-Sketch [17] using only output-directed manipulations, although the resulting programs for two of those four (iv, vi) encode an undesirable parameterization of the design.

For these four examples, Figure 15 lists the source lines of code (LOC) and math operations (MO) for the programs in both the present and prior version of Sketch-n-Sketch. The programs in the present work are shorter, largely because the drawing tools in [17] inserted a multi-line definition for each new shape, with named local variables for each parameter (cf. lines 20-23 of Figure 12). In this work, as a result of certain simplifications and generalizations of Sketch-n-Sketch internals, the “Standard Library Tools” in Figure 3 instead insert single-line calls to standard library functions, but these single line calls are comparable to the prior multi-line definitions. Consequently, LOC differences should not be taken as an indication of better code generation.

However, [17] explicitly mentioned a limitation that Sketch-n-Sketch sometimes inserted large math expressions too complicated for human comprehension. In particular, example (xi) Logo (via Three Tris) required 180 math operations in [17]. The present Sketch-n-Sketch requires only 11. The reduction in math operations comes from various improvements. Abandoning the strict left-top-right-bottom parameterization required by [17] means that common parameters such as a shape’s width no longer need to be computed. The Make Equal tool has also been improved: it attempts to reuse variables instead of inserting repeated math, and it relies on REDUCE [16] to simplify equations. Finally, the new offset widgets (introduced later in § 4.3) allow the programmer to pre-encode a natural parameterization of the design, creating a skeleton of needed and variables that later shapes can use for their positioning.

Programs for the remaining 11 of the 16 examples in Figure 15 can be created without text edits in the present system but not in [17] because they require features unique to this work. We now highlight these features by discussing the key steps in the creation of the first three examples in Figure 15.

4.2 Example (i): Koch Snowflake

The lambda logo example introduced Sketch-n-Sketch’s basic Draw-Relate-and-Abstract workflow as well as offered a glimpse into the novel contributions of this work: focused editing (to draw the border within the logo function), manipulable intermediate execution products (list widgets to select the group), and new tools (refactoring the function arguments). To more directly emphasize the capabilities of these novel techniques, here we demonstrate the construction of a von Koch fractal snowflake (Figure 1, Figure 15i). In particular, the example requires the manipulation of intermediates—throughout the majority of its construction the program will have no output—and focused editing is needed to build the recursive function. Finally, to highlight the expressive power of the Sketch-n-Sketch toolset, this example is constructed entirely via manipulations on the canvas. (Appendix A includes a video that demonstrates this example workflow with slight differences.)

Figure 16:
Motif for Koch fractal.

The design is created by recursively repeating the motif shown in Figure 16. Points labeled are inputs; points labeled will be placed and of the way between the inputs. A final point (labeled ) will be placed equidistant from the latter, forming an equilateral triangle. Repeating the motif between pairs of points will recursively define the fractal.

This motif requires two helper functions not provided in the standard library, so constructing the Koch fractal proceeds in four phases.

  1. We construct a helper function that, given two points, computes a point of the way between them (Figure 17). This function, reused backwards, will produce the point.

  2. We also create a function that, given two points, computes a third point that completes the equilateral triangle with its two input points (Figure 19).

  3. We use these two helpers to build a function that creates the motif and recursively repeats the motif within itself. This forms the points of a Koch curve, i.e. one side of the final snowflake (Figure 21).

  4. We lay out three instances of Koch curve points along the sides of an equilateral triangle, and then attach a polygon to the points to complete the snowflake design. (Figure 1).

Below, we discuss each of these steps in turn.

Example (i)a: One-Third Point Function

Tools introduced: Draw Point, Relate

Figure 17:
oneThirdPt function.

The first step is to create a function that, given two endpoints, returns a point of the way between them as shown in Figure 17. Creating this function utilizes several new Sketch-n-Sketch features: the Draw Point tool, point widgets, and the Relate tool.

The “Point or Offset” tool in the toolbox (Figure 3) lets the programmer click on the canvas to insert bare point definitions. Unlike shape drawing tools, these definitions are not added to the final shape list.

[x3, y3] as point3 = [264, 131]
[x2, y2] as point2 = [141, 179]
[x, y] as point = [87, 206]
svg (concat [

Whenever the evaluator encounters a number-number pair, a point widget for that coordinate is produced as a side effect. Thus, although the program’s output is technically empty, these point widgets still appear on the canvas for manipulation or for selection.

Figure 18: Relate tool results.

In this case, we move the points into roughly the positions we want (approximating Figure 17), select the points, and invoke the Relate tool. The Relate tool uses enumerative search (“guess-and-check”) to attempt to discover an arithmetic expression that, were it substituted into the program, would define one of the points in terms of the others and leave our points in roughly the same place. After about 20 seconds, three possible results become available in the Relate tool submenu (Figure 18).

The second result is mathematically equivalent to what we might deduce by hand (e.g. , and similarly for ). Choosing that result rewrites the point2 definition using the discovered arithmetic terms. Note also that the point definition is moved upward so its x and y variables are in scope for the inserted math.

[x3, y3] as point3 = [264, 131]
[x, y] as point = [87, 206]
[x2, y2] as point2 = [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]

Since this work reimagined the Abstract tool to simply abstract an expression over its free variables, the Abstract tool is no longer limited to building functions that return shapes as it was in [17]. Consequently, we select the point on the canvas and invoke the Abstract tool to turn the math into a reusable function. As in the lambda logo example, we can Rename the function on the canvas; here we choose the name oneThirdPt. Since the function is now complete (shown below) we select and Delete the three concrete points used to build the function—the function itself is left in our program. Since this function takes two points as input, it is drawable and appears in the “User-Defined Tools” section of the toolbox (Figure 1), which will prove advantageous later in the construction.

oneThirdPt [x3, y3] [x, y] =
  [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]

Example (i)b: Equi-Tri Point Function

Figure 19:
equiTriPt function.

Tools introduced: Distance Features

For the second step in the creation of our Koch fractal, we need a helper function that takes in two points and returns a point equidistant from both, forming an equilateral triangle (Figure 19).

The construction of this helper function proceeds in the same manner as the previous, except that the math for enforcing equidistance is too complicated for the Relate tool to discover by guess-and-check. Instead, after placing the three example points on the canvas, we explicitly select the distances between by clicking the distance features (the pink lines in Figure 20) that appear between all selected points (shown in green).

Figure 20:
Distance features between selected points.

We invoke the Make Equal tool to attempt to equalize the distances. An external solver (REDUCE [16]) discovers appropriate mathematical formulae to enforce this equality. Because there are many options for which items might be defined in terms of the others, Make Equal offers a large number of solutions. Hovering and previewing the first option, we see that it produces an extraneous offset widget (about which we defer discussion until the next example below), so we choose the second.

After abstracting the math and renaming the function to equiTriPt, we end up with the function below, which also becomes available as a new tool in the toolbox.

equiTriPt [x3, y3] [x2, y2] =
  [ (x2 + x3 + sqrt 3! * (y2 - y3))/ 2!, (y2 + y3 - sqrt 3! * (x2 - x3)) / 2!]

Example (i)c: Recursive Koch Curve Points Function

Tools introduced: Focused Recursive Drawing, Add to Output, Reorder in List, Choose Termination Condition

Figure 21:
makeKochPts function.

The third and main step of the construction is to build the fractal motif and then repeat the motif inside itself (Figure 21).

The motif is constructed using the two helper functions in our toolbox. Drawing the oneThirdPt function on the canvas creates our point along with two endpoints:

oneThirdPt2 = oneThirdPt [65, 199] [235, 141]

The point is obtained by snap-drawing another copy of the function backwards between those endpoints (shown mid-draw in Figure 22). Switching to the equiTriPt tool and snap-drawing between the point and point yields the final point of the motif. The code so far is shown below:

Figure 22:
Snap-draw oneThirdPt backwards to place point.
point = [65, 199]
point2 = [235, 141]
oneThirdPt2 = oneThirdPt point point2
oneThirdPt3 = oneThirdPt point2 point
equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2

With this last point selected (equiTriPt2), invoking Abstract yields the beginning of a function for drawing a Koch curve, which we Rename to makeKochPts. Recall that variables used only inside of the abstracted expression are heuristically pulled into the new function (with the exception of definitions that themselves have no free variables—there has to be something left to abstract over!). The resulting function takes in two endpoints (orange in Figure 23), calculates the and points (white in Figure 23), and outputs the protruding equiTriPt (blue in Figure 23).

makeKochPts point point2 =
  let oneThirdPt2 = oneThirdPt point point2 in
  let oneThirdPt3 = oneThirdPt point2 point in
  equiTriPt oneThirdPt3 oneThirdPt2
Figure 23:
The initial motif function, shown focused. Focused functions show inputs in orange and outputs in blue.

To make this function recursively repeat the motif, with the function focused (Figure 23) we draw the function inside itself between the leftmost two points. Sketch-n-Sketch inserts an if-then-else recursive skeleton and the recursive call:

makeKochPts point point2 =
  let oneThirdPt2 = oneThirdPt point point2 in
  let oneThirdPt3 = oneThirdPt point2 point in
  let equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 in
  if ??terminationCondition then
    let makeKochPts2 = makeKochPts point oneThirdPt3 in

To avoid infinite recursion, the guard expression ??terminationCondition—which is syntactic sugar for the value hole —returns false the first time the function is encountered in the call stack, and true if the function has appeared earlier in the call stack, affecting termination at a fixed depth of two. The programmer may run and manipulate the program, deferring selection of a termination condition until later.

Drawing makeKochPts between the remaining three pairs of points inserts the rest of the desired calls into the recursive branch.

  if ??terminationCondition then
    let makeKochPts2 = makeKochPts point oneThirdPt3 in
    let makeKochPts3 = makeKochPts oneThirdPt3 equiTriPt2 in
    let makeKochPts4 = makeKochPts equiTriPt2 oneThirdPt2 in
    let makeKochPts5 = makeKochPts oneThirdPt2 point2 in
Figure 24:
Initial recursive function with the motif repeated four times. Currently, only the blue point is an output.

The design looks like a fractal now (Figure 24), but still only one point is being output from the function—the white points in Figure 24 are only intermediates. Moreover, most of those intermediates are inside calls to the base case of our function—but we are focused on the recursive case. To modify the output of the base case, we click the call widget of the leftmost call to the base case (the inner light gray border in Figure 24) to focus that call instead (Figure 25).

Figure 25:
The focused base case.

We want the leftmost four points in the base case to be returned together as a list, i.e. we want the leftmost four points in Figure 25 to appear blue. The fifth point will be provided by the neighboring call to the base case.

After selecting the three additional points that should be in the output, we invoke Add to Output from the Output Tools menu. In order to output multiple points, the function must now return a list of points. The return expressions of both branches of our function are therefore wrapped in lists, and the three selected points are added to the list in the base case.

  if ??terminationCondition then
    [equiTriPt2, point, oneThirdPt3, oneThirdPt2]

The points must be in left-to-right order so that, when attached to a polygon, the design appears correctly. Selections in Sketch-n-Sketch are ordered, so our three added points are in the proper order (assuming we selected them in left-to-right order) but the pre-existing equiTriPt2 needs to be third in the list instead of at the head. The Reorder in List tool allows us to change its position in the list after selecting the point on the canvas—moving the point backwards twice puts it in the proper spot. The base case is now correct.

The recursive branch needs to be modified so that it returns a flattened list of all the points from the four calls to the base case. After re-focusing our editing on the recursive case instead of the base case, we repeat the same workflow: we select the four lists returned from the base cases (by selecting their list widgets) and invoke Add to Output. Sketch-n-Sketch realizes we are trying to combine lists together and inserts a concat (flatten) call so the function produces a list of points, rather than a list of lists of points. An [equiTriPt] singleton list from the original return value of the function is extraneously present in the final return list; it can be deleted after selecting its list widget. Thereafter we are left with the desired return expression with all the points:

      concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5]

What remains is to choose a termination condition. The focused call widget for makeKochPts displays the conditional not <| ??terminationCondition for the recursive case (upper right of Figure 24) which we can click to Choose Termination Condition. Currently, Sketch-n-Sketch offers only one kind of automatically generated termination: fixed depth. A depth argument is added to our makeKochPts function which is decremented on the recursive calls—these changes are visible in the final code (Figure 1). Additionally, the original example call to our function is given a depth of 2 with a {1-5} range annotation so that Sketch-n-Sketch draws a slider on the canvas for modifying the depth [8].

equiTriPt2 = makeKochPts 2{1-5} point point2

Example (i)d: Koch Snowflake Polygon

Tools introduced: Attach Polygon to Points

Figure 26:
Reusing equiTriPt for the sides of the snowflake.

Equipped with the makeKochPts function, which produces points for one side of the Koch snowflake, we use the equiTriPt helper function to draw a triangle (Figure 26), along the sides of which we can draw makeKochPts two more times.

The Group tool can be used to gather the three lists of points from the calls to makeKochPts into a single list. Group offers an option to concat (flatten) the lists rather than create a list of lists—this is the option we choose. To finish the design, we choose the “Polygon” tool and click the list widget for our group, thereby invoking Attach Polygon to Points. To polish the code, we select and equalize the three depth sliders for the three calls to makeKochPts, obtaining a single depth variable (and slider), and then perform a bit of renaming to result in code shown in Figure 1. After setting the depth to three with the slider, we find the menu option to hide all the widgets and obtain the final snowflake shown in Figure 1.

4.3 Example (ii): Tree Branch

Tools introduced: Draw Offset, Repeat Over List

To introduce offset widgets and the repetition tools, we discuss portions of the construction of the tree branch design shown in Figure 15ii. The overarching strategy is to build a rhombus abstracted over its center point which is then repeated over a list of points to form the leaves. For this example and the following, we omit mentioning uses of Rename.

Figure 27:
An offset widget.

The rhombus is constructed around a central point using offsets, which are simple additions or subtractions from an or coordinate. The “Point or Offset” tool in the toolbox (Figure 3) affects Draw Offset when dragged on the canvas, inserting an addition or subtraction operation into the program, e.g. xOffset = x + 102, which produces a corresponding widget on the canvas (Figure 27). If the new offset is not drawn from an existing point, a starting point is inserted into the program as well.

Offset amounts may snap to each other while drawing. If we draw a second offset of the same length in the opposite direction (Figure 28) a variable is inserted for the offset amount:

Figure 28:
Two offsets of the same amount.
num = 102
xOffset = x + num
xOffset2 = x - num
Figure 29:
Rhombus skeleton created with offsets.

Amount snapping enables us to quickly create the skeleton of the leaf rhombus (Figure 29). We then draw a polygon snapped to the offset endpoints and Abstract the resulting rhombus into a function parameterized over [x, y], halfW, and halfH. We will attach instances of this function over the branch.

The branch construction is shown in Figure 30. On top of a brown polygon two “deadspace” offsets are drawn inwards from the ends of the branch to form the start and end locations of the attachment points for the leaves. Between these locations we draw a function that returns a list of points, namely the pointsBetweenSepBy standard library function (Figure 3) that returns points separated from their neighbors by a fixed distance. With this function, making our branch longer will add more leaves rather than spacing them out.

Figure 30:
Progression of adding leaves to the branch. Top: Offsets for the leafless deadspace at the branch ends. Middle: List of points drawn between the deadspaces. Bottom: Rhombuses repeated over those points.

Finally, to repeat our leaf rhombus over the points, we first select the one copy of the rhombus on the canvas. The Output Tools panel then offers multiple tools for repeating the shape. We may Repeat With Function, repeating the shape over a new call to any one of the point-list-producing functions available, or we can Repeat Over List, repeating the shape over an existing point list in our program. Invoking Repeat Over List over the attachment points we just drew creates a new function abstracted over just a single point (rhombusFunc2 below) and maps that function over our leafAttachmentPts, completing our leafy branch.

rhombusFunc2 ([x, y] as point) =
  let halfW = 40 in
  let halfH = 83 in
  rhombusFunc point halfW halfH
repeatedRhombusFunc2 =
  map rhombusFunc2 leafAttachmentPts

4.4 Example (iii): Target

Figure 31: Target.

Tools introduced: Repeat by Indexed Merge, Fill Hole

Repeating over a point list allows copies of shapes to vary in their spacial positions, but not over any other attributes such as size or color. To support other repetition scenarios where the varying attributes could be calculated from an index (i.e. ), we offer a workflow which we now briefly demonstrate by the construction of a target (Figure 15iii).

To start, we draw three concentric circles snapped to the same center point and change the color of the middle circle (Figure 31). After selecting the three circles, we invoke Repeat by Indexed Merge, from which we select the second of two results (which differs from the first only in that it adds the reverse call seen below, so that i=0 always for the smallest circle).

circles =
  map (\i ->
      circle ??(1 => 0, 2 => 466, 3 => 0) point ??(1 => 114, 2 => 68, 3 => 25))
    (reverse (zeroTo 3{0-15}))

The program maps an anonymous function that takes an index (\i -> ...) over the list [2,1,0]; each index is thus transformed into one of our circles. The anonymous function is a syntactic merger of our original three circle definitions. The differences between the expressions—radius and color—have been turned into value holes (§ 3.2). The omitted flag on the value hole defaults to false, such that e.g. the first hole evaluates to 0 the first time it is encountered, and then 466, and then 0.

Figure 32: Options to fill holes.

Sketch-n-Sketch employs sketch-based synthesis [40] to resolve value holes. As shown in Figure 32, all suggested fillings involve i, the only variable that differs in the execution environments. For the first hole, we choose the mod i 2 == 0! conditional to obtain alternating colors. For the second we choose the only option, a expression, to calculate the radii of our circles. Similar to the Koch snowflake example, the inserted code zeroTo 3{0-15} contains a slider annotation allowing us to manipulate the number of circles. We display five circles for the final design (Figure 15iii).

4.5 Limitations: Remaining Tasks

Several additional features would be needed for our system to construct all the examples from [17] and Watch What I Do: Programming by Demonstration [1] without text edits.

Prior Sketch-n-Sketch [17].

Besides the basic lambda logo (§ 2 and § 4.1) and the lambda logo constructed with triangles (Figure 15xi), [17] presented three additional examples, although each required text edits to reach their most developed form. Various features are required to create these examples solely via output-directed interactions.

Two of the remaining examples in [17] utilize paths (“Garden Logo” and “Coffee Mug”). For the tasks we attempted in this work, the path tool was not needed and so was disabled pending internal updates to allow path drawing to work with snaps. Additionally, as the Group tool no longer performs domain-specific rewriting, a new Rewrite as Offsets tool would be useful to rewrite the coordinates of selected points so they are computed relative to some anchor point. “Garden Logo” would additionally benefit from special tooling for mirrored drawing across a vertical symmetry line (a mirrored design can be created with offsets and snaps in the present work, but it’s tedious).

The third remaining example (“Snip Polygon”) relied on special tooling in [17] to draw a polygon with its points positioned relative to a bounding box. This tooling was removed in the present work as we discarded the pervasive bounding box parameterization and reduced reliance on hard-coded templates. Equivalent tooling would have to be restored (perhaps a Rewrite Relative to Bounds tool). Additionally, for the example to reach its fully developed form without text edits, Sketch-n-Sketch would need an output-directed mechanism for specifying min/max calculations.

Watch What I Do: Programming by Demonstration [1].

Of the 15 tasks in WWID: PBD that may be interpreted as parametric drawings, our system is able to complete 4.

Two of the remaining WWID: PBD tasks can be partially completed in this work (Figure 15vii and Figure 15viii). To complete these tasks, the “Box Volume” example would require an interaction to compute and display the numeric volume of the folded box. The “Xs” task would require more precise control over what definitions are pulled into the abstraction. (Not all uses of a squareWidth parameter are pulled in to the abstraction, causing the design to render incorrectly if we try to draw an X with a different square size.)

The remaining nine tasks are diverse; no single feature would help with more than two or three. One prominent missing feature is the ability to draw text boxes, with other elements placed relative to the text size. Beyond this, several examples require various list operations. Sketch-n-Sketch would also need to reason about intersections of lines with shape edges, to offer ways to specify overlapping and containment-based constraints, and to solve such constraints simultaneously. Finally, one example would require Sketch-n-Sketch to create if-then-else branches outside of a recursive or hole-filling setting.

5 Related Work

Many related approaches share one or both characteristics of our work, providing direct manipulation mechanisms to (a) transform programs and (b) build parametric designs.

Parametric Computer-Aided Design (CAD).

Feature-based parametric CAD systems record user actions as a series of steps, which can be reused. Among CAD systems, EBP [34] is notable for offering a PBD workflow to create loops and conditionals.

Drawing With Constraints.

Several systems integrate constraint specification into the visual design process (e.g. [44, 21]). Apparatus [36], Recursive Drawing [37], and Geometer’s SketchPad [20] are notable for supporting recursion.

Constraint-Oriented Programming (COP).

Other constraint-oriented systems, following in the footsteps of SketchPad [41], explicitly view building a constrained system as a programming task [5, 19, 13]. While offering varying degrees of visibility into the code, these systems are distinguished by running constraint solvers alongside the program.

Programming by Demonstration (PBD).

Several PBD approaches [9] use shape drawing as a domain for exploring non-textual programming techniques. These systems usually rely on a visual representation of the program rather than a textual one (e.g. [22, 24]), or show actions step-by-step [27]. We also use shape drawing as our application domain, but we do not hide the program text. Although not as visual as peer PBD systems, Tinker [25] is notable for supporting recursion by demonstration; indeed, any Lisp expression may be created, albeit via manipulations performed on a symbolic representation not far removed from the underlying Lisp. PBD techniques have recently been developed for data visualization [42], mobile applications for collaboration [11], web scraping [3, 6], and API discovery [45].

Output-Directed Programming.

A number of recent systems offer a regular text-based programming experience augmented with abilities to directly manipulate output to enact code changes. The transformations available may be “small” (such as changing constants [8, 23, 28], strings [43, 39, 23, 28], or list literals [28]), but several systems enable “larger” program changes via output manipulation. One is the prior version of Sketch-n-Sketch [17] upon which we build; § 4.1 provides a comparison. We discuss two more systems below.

Transmorphic [38] re-implements the Morphic UI framework [26] using static, functional (i.e. stateless) views. Transmorphic retains Morphic’s ability to directly manipulate shapes by affecting changes to the view’s (text-based) code rather than changing live object state. Adding and removing shapes, or changing a shape’s primitive properties, are supported.

Like Sketch-n-Sketch, APX [31, 30] is a two-pane (code box and output canvas) environment for creating shape-drawing programs; APX additionally supports real-time visual simulations. Numbers in the program can be changed by directly manipulating shape attributes. Some larger changes—e.g. grouping and insertion of new shapes—are supported, but most interactions focus on refactoring by directly manipulating terms in the code box.

6 Conclusion and Future Work

We presented new techniques for the output-directed SVG programming environment Sketch-n-Sketch and used the system to construct a variety of non-trivial programs entirely through direct manipulation. Our ultimate goal, however, is not to create a practical drawing tool but to use this particular application domain as a laboratory for advancing the expressiveness of output-directed programming for a variety of future programming settings. There are several natural avenues to explore in this longer-term direction.

Customizing Widgets.

Mechanisms for customization may help address the open question of how to render graphical representations of program evaluation and of domain objects that are not inherently visual. One approach would be to design an API for “toSvg” functions that specify how to render graphical representations of intermediate data structures. Analogous to “toString” functions that render text, a toSvg API might similarly render composite data structures via recursively calling toSvg for each of the structure’s elements.

Synthesizing Program Transformations.

The building blocks in our implementation—provenance, value holes, and roles—have allowed us to implement a variety of program transformations, each of which has been hand-coded one-by-one. This effort is time-consuming, error-prone, and limits the composability of transformations. Program synthesis techniques have been successfully incorporated into refactoring tasks for class-based languages [35]; perhaps such techniques can be extended to streamline the specification of output-directed transforms that operate simultaneously on both abstract and concrete syntax.

User Interactions for Deciphering Human Intent.

There are often multiple valid program changes associated with a specific user action. To resolve ambiguity, our system displays code and output differences and asks the user to choose. Richer interactions are needed to explain the differences (e.g., change impact analysis [14, 10]) and to better resolve the user’s intent (e.g., by asking additional questions [29]). It may be also useful to propagate sets of program changes (rather than committing to one after each action), because subsequent user interactions may help resolve ambiguities during earlier phases—a key challenge will be to render unified representations and offer manipulations despite variations among candidate programs. In order for the live and immediate experience afforded by output-directed programming to scale into usable systems for different programming tasks and user scenarios, developing and refining such user interaction techniques remain crucial challenges to tackle in future work.


  • [1] A Test Suite for Programming by Demonstration. In Richard Potter and David Maulsby, editors, Watch What I Do: Programming by Demonstration. MIT Press, 1993.
  • [2] Umut A. Acar, Amal Ahmed, James Cheney, and Roly Perera. A Core Calculus for Provenance. Journal of Computer Security, 2013.
  • [3] Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. Ringer: Web Automation by Demonstration. In Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2016.
  • [4] Gilbert Louis Bernstein and Wilmot Li. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. Transactions on Graphics (TOG), 2015.
  • [5] Alan Borning. The Programming Language Aspects of ThingLab. Transactions on Programming Languages and Systems (TOPLAS), October 1981.
  • [6] Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. Rousillon: Scraping Distributed Hierarchical Web Data. In Symposium on User Interface Software and Technology (UIST), 2018.
  • [7] Salman Cheema, Sumit Gulwani, and Joseph LaViola. QuickDraw: Improving Drawing Experience for Geometric Diagrams. In Conference on Human Factors in Computing Systems (CHI), 2012.
  • [8] Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. Programmatic and Direct Manipulation, Together at Last. In Conference on Programming Language Design and Implementation (PLDI), 2016.
  • [9] Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, editors. Watch What I Do: Programming by Demonstration. MIT Press, 1993.
  • [10] Bogdan Dit, Michael Wagner, Shasha Wen, Weilin Wang, Mario Linares Vásquez, Denys Poshyvanyk, and Huzefa H. Kagdi. ImpactMiner: A Tool for Change Impact Analysis. In International Conference on Software Engineering, Companion Proceedings (ICSE-C), 2014.
  • [11] Jonathan Edwards, Jodie Chen, and Alessandro Warth. Live End-User Programming. In LIVE Workshop, 2016.
  • [12] Evan Czaplicki. Elm. http://elm-lang.org.
  • [13] Tim Felgentreff, Alan Borning, Robert Hirschfeld, Jens Lincke, Yoshiki Ohshima, Bert Freudenberg, and Robert Krahn. Babelsberg/JS - A Browser-Based Implementation of An Object Constraint Language. In European Conference on Object-Oriented Programming (ECOOP), 2014.
  • [14] Malcom Gethers, Bogdan Dit, Huzefa Kagdi, and Denys Poshyvanyk. Integrated Impact Analysis for Managing Software Changes. In International Conference on Software Engineering (ICSE), 2012.
  • [15] Michael Gleicher and Andrew Witkin. Drawing with Constraints. The Visual Computer: International Journal of Computer Graphics, 1994.
  • [16] Anthony C. Hearn. REDUCE: A User-Oriented Interactive System for Algebraic Simplification. In Interactive Systems for Experimental Applied Mathematics. Academic Press, 1968.
  • [17] Brian Hempel and Ravi Chugh. Semi-Automated SVG Programming via Direct Manipulation. In Symposium on User Interface Software and Technology (UIST), 2016.
  • [18] Brian Hempel, Justin Lubin, Grace Lu, and Ravi Chugh. Deuce: A Lightweight User Interface for Structured Editing. In International Conference on Software Engineering (ICSE), 2018.
  • [19] Allan Heydon and Greg Nelson. The Juno-2 Constraint-Based Drawing Editor. In Technical Report 131a, Digital Systems Research, Digital Equipment Corporation, 1994.
  • [20] R. Nicholas Jackiw and William F. Finzer. The Geometer’s Sketchpad: Programming by Geometry. In Watch What I Do: Programming by Demonstration. MIT Press, 1993.
  • [21] Jennifer Jacobs, Sumit Gogia, Radomír Mech, and Joel R. Brandt. Supporting Expressive Procedural Art Creation Through Direct Manipulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017., 2017.
  • [22] David Kurlander. Graphical Editing by Example. PhD thesis, Columbia University, 1993.
  • [23] Kevin Kwok and Guillermo Webster. Carbide Alpha. https://alpha.trycarbide.com/, 2016.
  • [24] Henry Lieberman. Mondrian: A Teachable Graphical Editor. In Watch What I Do: Programming by Demonstration. MIT Press, 1993.
  • [25] Henry Lieberman. Tinker: A Programming By Demonstration System for Beginning Programmers. In Watch What I Do: Programming by Demonstration. MIT Press, 1993.
  • [26] John H. Maloney and Randall B. Smith. Directness and Liveness In the Morphic User Interface Construction Environment. In Symposium on User Interface Software and Technology (UIST), 1995.
  • [27] David L. Maulsby, Ian H. Witten, and Kenneth A. Kittlitz. Metamouse: Specifying Graphical Procedures by Example. In Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), 1989.
  • [28] Mikaël Mayer, Viktor Kunčak, and Ravi Chugh. Bidirectional Evaluation with Direct Manipulation. Proceedings of the ACM on Programming Languages (PACMPL), Issue OOPSLA, 2018.
  • [29] Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software and Technology (UIST), 2015.
  • [30] McDirmid, Sean. The Future of Programming will be Live. Curry On 2016. https://www.youtube.com/watch?v=bnqkglrSqrg.
  • [31] McDirmid, Sean. A Live Programming Experience. Future Programming Workshop, Strange Loop 2015. https://onedrive.live.com/download?cid=51C4267D41507773&resid=51C4267D41507773%2111492&authkey=AMwcxdryTyPiuW8 https://www.youtube.com/watch?v=YLrdhFEAiqo.
  • [32] Dan R. Olsen, Jr. Evaluating User Interface Systems Research. In Symposium on User Interface Software and Technology (UIST), 2007.
  • [33] Roly Perera, Umut A. Acar, James Cheney, and Paul Blain Levy. Functional Programs That Explain Their Work. In ACM SIGPLAN International Conference on Functional Programming, ICFP’12, Copenhagen, Denmark, September 9-15, 2012, 2012.
  • [34] Guy Pierra, Jean-Claude Potier, and Patrick Girard. The EBP System: Example Based Programming System for Parametric Design. In Modelling and Graphics in Science and Technology. Springer Berlin Heidelberg, 1996.
  • [35] Veselin Raychev, Max Schäfer, Manu Sridharan, and Martin Vechev. Refactoring with Synthesis. In Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2013.
  • [36] Schachman, Toby. Apparatus. http://aprt.us/.
  • [37] Schachman, Toby. Recursive Drawing. Master’s thesis, New York University Interactive Telecommunications Program, 2012. http://recursivedrawing.com/.
  • [38] Robin Schreiber, Robert Krahn, Daniel H. H. Ingalls, and Robert Hirschfeld. Transmorphic: Mapping Direct Manipulation to Source Code Transformations. 2016.
  • [39] Christopher Schuster and Cormac Flanagan. Live Programming by Example: Using Direct Manipulation for Live Program Synthesis. In LIVE Workshop, 2016.
  • [40] Armando Solar-Lezama. Program Synthesis by Sketching. PhD thesis, UC Berkeley, 2008.
  • [41] Ivan Sutherland. Sketchpad, A Man-Machine Graphical Communication System. PhD thesis, MIT, 1963.
  • [42] Victor, Bret. Drawing Dynamic Visualizations. http://worrydream.com/#!/DrawingDynamicVisualizationsTalk.
  • [43] Xiaoyin Wang, Lu Zhang, Tao Xie, Yingfei Xiong, and Hong Mei. Automating Presentation Changes in Dynamic Web Applications via Collaborative Hybrid Analysis. In International Symposium on the Foundations of Software Engineering (FSE), 2012.
  • [44] Haijun Xia, Bruno Araújo, Tovi Grossman, and Daniel J. Wigdor. Object-Oriented Drawing. In Conference on Human Factors in Computing Systems (CHI), 2016.
  • [45] Kuat Yessenov, Ivan Kuraj, and Armando Solar-Lezama. DemoMatch: API Discovery from Demonstrations. In Conference on Programming Language Design and Implementation (PLDI), 2017.

Appendix A Note About Supplementary Materials

Our Supplementary Materials include:

  1. Our Sketch-n-Sketch browser-based application.

  2. An unnarrated screencast to accompany the Lambda Logo workflow described in § 2.

  3. A narrated screencast to accompany the Koch Snowflake flow described in § 4.2. Please note that the video shows a slightly older version of our system that requires a few extra Reorder in List operations.

Appendix B Full Code Listings

Verbatim code listings as produced by Sketch-n-Sketch, albeit with occasional comments added at top.

b.1 (i) Koch Snowflake

1-- Final as in paper, but depth 2
4equiTriPt [x3, y3] [x2, y2] =
5  [ (x2 + x3 + sqrt 3! * (y2 - y3))/ 2!, (y2 + y3 - sqrt 3! * (x2 - x3)) / 2!]
7oneThirdPt [x3, y3] [x, y] =
8  [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]
10point = [39, 314]
12point2 = [490, 301]
14makeKochPts depth point point2 =
15  let oneThirdPt2 = oneThirdPt point point2 in
16  let oneThirdPt3 = oneThirdPt point2 point in
17  let equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 in
18  if depth < 2 then
19    [point, oneThirdPt3, equiTriPt2, oneThirdPt2]
20  else
21    let makeKochPts2 = makeKochPts (depth - 1) point oneThirdPt3 in
22    let makeKochPts3 = makeKochPts (depth - 1) oneThirdPt3 equiTriPt2 in
23    let makeKochPts4 = makeKochPts (depth - 1) equiTriPt2 oneThirdPt2 in
24    let makeKochPts5 = makeKochPts (depth - 1) oneThirdPt2 point2 in
25      concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5]
27depth = 2{1-5}
29topPts = makeKochPts depth point point2
31botCorner = equiTriPt point2 point
33rightPts = makeKochPts depth point2 botCorner
35leftPts = makeKochPts depth botCorner point
37snowflakePts = concat [topPts, rightPts, leftPts]
39polygon1 =
40  let pts = snowflakePts in
41  let [color, strokeColor, strokeWidth] = [124, 360, 2] in
42    polygon color strokeColor strokeWidth pts
44svg (concat [
45  [polygon1]

b.2 (ii) Tree Branch

2[branchLeft, branchY] as branchAnchorPt = [53, 542]
4rhombusFunc [x, y] halfW halfH =
5  let xOffset = x + halfW in
6  let xOffset2 = x - halfW in
7  let yOffset = y - halfH in
8  let yOffset2 = y + halfH in
9  let pts = [[x, yOffset], [xOffset, y], [x, yOffset2], [xOffset2, y]] in
10  let [color, strokeColor, strokeWidth] = [121, 360, 2] in
11    polygon color strokeColor strokeWidth pts
13rhombusFunc2 ([x, y] as point) =
14  let halfW = 40 in
15  let halfH = 83 in
16  rhombusFunc point halfW halfH
18branchHalfW = 48
20branchTop = branchY - branchHalfW
22branchBot = branchY + branchHalfW
24branchRight = branchLeft + 405
26branch =
27  let pts = [[branchLeft, branchTop], [branchRight, branchY], [branchLeft, branchBot]] in
28  let [color, strokeColor, strokeWidth] = [29, 360, 2] in
29    polygon color strokeColor strokeWidth pts
31deadspace = 72
33leafAttachmentStartX = branchLeft + deadspace
35leafAttachmentEndX = branchRight - deadspace
37leafAttachmentPts = pointsBetweenSepBy [leafAttachmentStartX, branchY] [leafAttachmentEndX, branchY] 100
39leaves =
40  map rhombusFunc2 leafAttachmentPts
42svg (concat [
43  [branch],
44  leaves

b.3 (iii) Target

2point = [236, 241]
4circles =
5  map (\i ->
6      circle (if mod i 2! == 0! then 0 else 466) point (22 + i * 46))
7    (reverse (zeroTo 5{0-15}))
9svg (concat [
10  circles

b.4 (iv) Precisions Floor Plan

2point = [82, 136]
4h = 239
6w = 444
8floorRect = rect 36 point w h
10tableRect = rect 188 point (w / 3!) h
12svg (concat [
13  [floorRect],
14  [tableRect]

b.5 (v) Mondrian Arch

2[left, top] as topLeft = [95, 171]
4height = 335
6stoneWidth = 73
8width = 325
10archFunc ([left, top] as topLeft) width height stoneWidth =
11  let lintel = rect 210 topLeft width stoneWidth in
12  let pillarHeight =height - stoneWidth in
13  let pillarTop = top + stoneWidth in
14  let leftPillar = rect 0 [left, pillarTop] stoneWidth ( pillarHeight) in
15  let rightPillar = rect 134 [ left + width- stoneWidth, pillarTop] stoneWidth pillarHeight in
16  [lintel, leftPillar, rightPillar]
18arch = archFunc topLeft width height stoneWidth
20svg (concat [
21  arch

b.6 (vi) Balance Scale

2[centerX, pillarTop] as point2 = [236, 147]
4y3 = 181
6[x, y] as point = [89, 231]
8[x3, y3] as point3 = [noWidgets (sqrt (pow centerX 2! - 2! * centerX * x + pow x 2! - 2! * pillarTop * y + 2! * pillarTop * y3 + pow y 2! - pow y3 2!) + centerX), y3]
10trayWireWireFunc ([x, y] as topPoint) hangDistance =
11  let yOffset = y + hangDistance in
12  let [x1, y1] as point1 = [x, yOffset] in
13  let trayHalfW = 81 in
14  let left = x1 - trayHalfW in
15  let right = x + trayHalfW in
16  let tray = ellipse 40 point1 trayHalfW 30 in
17  let color = 434 in
18  let strokeWidth = 5 in
19  let wire1 = line color strokeWidth topPoint [left, yOffset] in
20  let wire2 = line color strokeWidth topPoint [right, yOffset] in
21  [tray, wire1, wire2]
23baseCenter = [centerX, 496]
25color = 208
27pillar = line color 20 point2 baseCenter
29base = ellipse color baseCenter 109 33
31strokeWidth = 15
33leftArm = line color strokeWidth point2 point
35rightArm = line color strokeWidth point2 point3
37hangDistance = 171
39hangingTray1 = trayWireWireFunc point hangDistance
41hangingTray2 = trayWireWireFunc point3 hangDistance
43svg (concat [
44  [pillar],
45  [base],
46  [leftArm],
47  [rightArm],
48  hangingTray1,
49  hangingTray2

b.7 (vii) Box Volume

1[x9, y9] as point9 = [130, 562]
3[x, y] as point = [102, 109]
5bigW = 271
7[x1, y1] as point1 = [ x+ bigW, y]
9yOffset = y + bigW
11xOffset = x + bigW
13y1Offset = y1 + bigW
15cutW = 58
17yOffset2 = y + cutW
19xOffset2 = x + cutW
21[x2, y2] as point2 = [xOffset2, y]
23y2Offset = y2 + cutW
25yOffsetOffset = yOffset - cutW
27[x3, y3] as point3 = [x, yOffset]
29x3Offset = x3 + cutW
31[x4, y4] as point4 = [x3Offset, yOffset]
33y4Offset = y4 - cutW
35xOffsetOffset = xOffset - cutW
37[x5, y5] as point5 = [xOffset, y]
39y5Offset = y5 + cutW
41[x6, y6] as point6 = [xOffsetOffset, y]
43y6Offset = y6 + cutW
45[x7, y7] as point7 = [x1, y1Offset]
47x7Offset = x7 - cutW
49y1OffsetOffset = y1Offset - cutW
51[x8, y8] as point8 = [x7Offset, y1Offset]
53y8Offset = y8 - cutW
55color = 39
57topDownTemplate =
58  let pts = [[xOffset2, y2Offset], point2, point6, [xOffsetOffset, y6Offset], [xOffset, y5Offset], [x1, y1OffsetOffset], [x7Offset, y8Offset], point8, point4, [x3Offset, y4Offset], [x, yOffsetOffset], [x, yOffset2]] in
59  let [strokeColor, strokeWidth] = [360, 2] in
60    polygon color strokeColor strokeWidth pts
62baseW = bigW - 2! * cutW
64xOffset2Offset = xOffset2 + baseW
66x3OffsetOffset = x3Offset + baseW
68x9Offset = x9 + baseW
70y9Offset = y9 - cutW
72[x10, y10] as point10 = [x9Offset, y9]
74y10Offset = y10 - cutW
76y11 = 692
78num = 367
80onLine2 = onLine point9 [num, y11] (baseW / sqrt (pow x9 2! - 2! * x9 * num + pow num 2! + pow y9 2! - 2! * y9 * y11 + pow y11 2!))
82[x11, _] = onLine2
84fstOffset = x11 + baseW
86[_, y12] = onLine2
88[_, y13] = onLine2
90[x13, y13] as point13 = [fstOffset, y13]
92y13Offset = y13 - cutW
94x9OffsetY10OffsetPair = [x9Offset, y10Offset]
96xY9OffsetPair = [x9, y9Offset]
98boxBack =
99  let pts = [point9, xY9OffsetPair, x9OffsetY10OffsetPair, point10] in
100  let [color, strokeColor, strokeWidth] = [color, 360, 2] in
101    polygon color strokeColor strokeWidth pts
103boxBot =
104  let pts = [point9, point10, point13, onLine2] in
105  let [color, strokeColor, strokeWidth] = [color, 360, 2] in
106    polygon color strokeColor strokeWidth pts
108[x14, y14] as point14 = [fstOffset, y13Offset]
110x14Offset = x14 - baseW
112boxRight =
113  let pts = [point10, x9OffsetY10OffsetPair, point14, point13] in
114  let [color, strokeColor, strokeWidth] = [color, 360, 2] in
115    polygon color strokeColor strokeWidth pts
117x14OffsetY13OffsetPair = [x14Offset, y13Offset]
119boxLeft =
120  let pts = [xY9OffsetPair, x14OffsetY13OffsetPair, onLine2, point9] in
121  let [color, strokeColor, strokeWidth] = [color, 360, 2] in
122    polygon color strokeColor strokeWidth pts
124boxFront =
125  let pts = [onLine2, x14OffsetY13OffsetPair, point14, point13] in
126  let [color, strokeColor, strokeWidth] = [color, 360, 2] in
127    polygon color strokeColor strokeWidth pts
129svg (concat [
130  [topDownTemplate],
131  [boxBack],
132  [boxBot],
133  [boxRight],
134  [boxLeft],
135  [boxFront]

b.8 (viii) Xs

2[x, y] as point = [198, 216]
4squareW = 66
6halfWidth =squareW / 2!
8n = 2{0-10}
10fill = 72
12squareByCenter2Func center2 =
13  squareByCenter fill center2 halfWidth
15fill2 = 218
17squareByCenter2Func2 center2 =
18  squareByCenter fill2 center2 halfWidth
20squareByCenter2Func3 center2 =
21  squareByCenter fill center2 halfWidth
23squareByCenter2Func4 center2 =
24  squareByCenter fill2 center2 halfWidth
26boxyXFunc ([x, y] as point) squareW n =
27  let xOffset = x + squareW in
28  let xOffset2 = x - squareW in
29  let yOffset = y - squareW in
30  let yOffset2 = y + squareW in
31  let squareByCenter1 = squareByCenter 426 point ( halfWidth) in
32  let ySep =0! - squareW in
33  let nPointsSepBy2 = nPointsSepBy n [xOffset, yOffset] squareW ( ySep) in
34  let repeatedSquareByCenter2Func =
35    map squareByCenter2Func nPointsSepBy2 in
36  let nPointsSepBy3 = nPointsSepBy n [xOffset, yOffset2] squareW squareW in
37  let repeatedSquareByCenter2Func21 =
38    map squareByCenter2Func2 nPointsSepBy3 in
39  let nPointsSepBy4 = nPointsSepBy n [xOffset2, yOffset2] ySep squareW in
40  let repeatedSquareByCenter2Func3 =
41    map squareByCenter2Func3 nPointsSepBy4 in
42  let nPointsSepBy5 = nPointsSepBy n [xOffset2, yOffset] ySep (0! - squareW) in
43  let repeatedSquareByCenter2Func4 =
44    map squareByCenter2Func4 nPointsSepBy5 in
45  let squareByCenterSingleton =
46    [squareByCenter1] in
47  concat [squareByCenterSingleton, repeatedSquareByCenter2Func, repeatedSquareByCenter2Func21, repeatedSquareByCenter2Func3, repeatedSquareByCenter2Func4]
49boxyX = boxyXFunc point squareW n
51boxyXFunc1 = boxyXFunc [506, 225] squareW 1{0-10}
53boxyXFunc2 = boxyXFunc [329, 666] squareW 3{0-10}
55svg (concat [
56  boxyX,
57  boxyXFunc1,
58  boxyXFunc2

b.9 (ix) Battery

1-- Simple Black Battery, abstracted b/c the point of the presentation in Lillicon is that you could make different versions for different icon sizes
3[x, y] as point = [66, 148]
5h4 = 141
7w = 274
9fill = 362
11h = 73
13batteryFunc ([x, y] as point) h4 w fill h =
14  let body = rect fill point w h4 in
15  let head = rect fill [ x+ w, (h4 - h + 2! * y) / 2!] 40 h in
16  [body, head]
18battery = batteryFunc point h4 w fill h
20svg (concat [
21  battery

b.10 (x) Ladder

1-- 2012 Quickdraw Fig. 1; cited as originally from a math text
2-- Trick is to draw offset first, then snap line to it exactly, then repeat the line.
4w = 126
6color = 366
8strokeWidth = 8
10line1Func ([x, y] as point) =
11  let xOffset = x + w in
12  line color strokeWidth point [xOffset, y]
14left = 104
16top = 119
18rungs =
19  map line1Func (nVerticalPointsSepBy 4{0-10} [left, top] 50)
21bot = 346
23leftLine = line color strokeWidth [left, top] [left, bot]
25rightLine = line color strokeWidth [ left+ w, top] [ left+ w, bot]
27svg (concat [
28  rungs,
29  [leftLine],
30  [rightLine]

b.11 (xi) Logo (via Three Tris)

1-- Need to draw the bot-right delta offsets before drawing the midpoint
2-- Because (ugh) getting the offsets to draw from the correct base points is hard.
3-- Abstracted (after grouping w/o gathering dependencies)
5[x, y] as point = [88, 104]
7w = 331
9 = 32
11lambdaFunc ([x, y] as point) w  leftColor botColor bigColor =
12  let yOffset = y + w in
13  let xOffset = x + w in
14  let [x1, y1] as point1 = [x, yOffset] in
15  let x1Offset = x1 +  in
16  let yOffsetOffset = yOffset -  in
17  let [x2, y2] as point2 = [xOffset, y] in
18  let y2Offset = y2 + w in
19  let [x3, y3] as point3 = [xOffset, y2Offset] in
20  let x3Offset = x3 -  in
21  let y2OffsetOffset = y2Offset -  in
22  let xOffset2 = x +  in
23  let yOffset2 = y +  in
24  let midpoint2 = midpoint point point3 in
25  let [x4, _] = midpoint2 in
26  let fstOffset = x4 -  in
27  let [_, y4] = midpoint2 in
28  let sndOffset = y4 +  in
29  let leftTri =
30    let [_, y5] = midpoint2 in
31    let pts = [[x, yOffset2], [fstOffset, y5], [x, yOffsetOffset]] in
32    let [color, strokeColor, strokeWidth] = [leftColor, 360, 2] in
33      polygon color strokeColor strokeWidth pts in
34  let botTri =
35    let [x5, _] = midpoint2 in
36    let pts = [[x1Offset, yOffset], [x5, sndOffset], [x3Offset, y2Offset]] in
37    let [color, strokeColor, strokeWidth] = [botColor, 360, 2] in
38      polygon color strokeColor strokeWidth pts in
39  let bigTri =
40    let pts = [[xOffset2, y], point2, [xOffset, y2OffsetOffset]] in
41    let [color, strokeColor, strokeWidth] = [bigColor, 360, 2] in
42      polygon color strokeColor strokeWidth pts in
43  [leftTri, botTri, bigTri]
45lambda = lambdaFunc point w  27 245 148
47svg (concat [
48  lambda

b.12 (xii) N Boxes

2boxes =
3  map (\i ->
4      rect 200 [ 50 + i * 76, 110] 55 195)
5    (zeroTo 7{0-15})
7svg (concat [
8  boxes

b.13 (xiii) Ferris Wheel

2point = [307, 334]
4r = 166
6attachmentPts = nPointsOnCircle 7{0-10} 0.06280000000000001{-3.14-3.14} point r
8color = 434
10spokeFunc point2 =
11  line color 5 point point2
13spokes =
14  map spokeFunc attachmentPts
16carFunc center2 =
17  squareByCenter 48 center2 25
19cars =
20  map carFunc attachmentPts
22capFunc point2 =
23  circle 364 point2 9
25caps =
26  map capFunc attachmentPts
28ring1 = ring color 7 point r
30hub = circle 362 point 44
32svg (concat [
33  [hub],
34  cars,
35  spokes,
36  [ring1],
37  caps

b.14 (xiv) Pencil Tip

2[taperStartX, y] as point = [253, 269]
4bodyHalfL = 118
6pencilHalfW = 61
8ratio = 0.62
10pencilFunc [taperStartX, y] pencilHalfW bodyHalfL taperL ratio =
11  let bodyCenterX = taperStartX - bodyHalfL in
12  let top = y - pencilHalfW in
13  let bot = y + pencilHalfW in
14  let tipX = taperStartX + taperL in
15  let body = rectByCenter 42 [bodyCenterX, y] bodyHalfL pencilHalfW in
16  let tipPt = [tipX, y] in
17  let taperStartTopPt = [taperStartX, top] in
18  let leadStartTopPt = onLine taperStartTopPt tipPt ratio in
19  let taperStartBotPt = [taperStartX, bot] in
20  let leadStartBotPt = onLine taperStartBotPt tipPt ratio in
21  let shavedWood =
22    let pts = [taperStartBotPt, taperStartTopPt, leadStartTopPt, leadStartBotPt] in
23    let [color, strokeColor, strokeWidth] = [460, 360, 0] in
24      polygon color strokeColor strokeWidth pts in
25  let lead =
26    let pts = [leadStartBotPt, leadStartTopPt, tipPt] in
27    let [color, strokeColor, strokeWidth] = [397, 360, 0] in
28      polygon color strokeColor strokeWidth pts in
29  [body, shavedWood, lead]
31pencil = pencilFunc point pencilHalfW bodyHalfL 205 ratio
33svg (concat [
34  pencil

b.15 (xv) Arrows

2pt2 = [405, 134]
4pt1 = [109, 238]