An Empirical Comparison of PDDL-based and ASP-based Task Planners Jiang and Stone are with the Department of Computer Science, The University of Texas at Austin. Zhang is with the Department of Electrical Engineering and Computer Science, Cleveland State University. Khandelwal is with Cogitai, Inc. This work was conducted when Zhang and Khandelwal were with The University of Texas at Austin.

An Empirical Comparison of Pddl-based and Asp-based Task Planners 1

Abstract

General purpose planners enable AI systems to solve many different types of planning problems. However, many different planners exist, each with different strengths and weaknesses, and there are no general rules for which planner would be best to apply to a given problem. In this paper, we empirically compare the performance of state-of-the-art planners that use either the Planning Domain Description Language (PDDL), or Answer Set Programming (ASP) as the underlying action language. PDDL is designed for automated planning, and PDDL-based planners are widely used for a variety of planning problems. ASP is designed for knowledge-intensive reasoning, but can also be used for solving planning problems. Given domain encodings that are as similar as possible, we find that PDDL-based planners perform better on problems with longer solutions, and ASP-based planners are better on tasks with a large number of objects or in which complex reasoning is required to reason about action preconditions and effects. The resulting analysis can inform selection among general purpose planning systems for a particular domain.

Keywords:
First keyword Second keyword More

1 Introduction

In a general purpose planning system, task planning problems are tackled by problem-independent solvers based on a description of the domain in a declarative language. Such systems are extremely useful in application domains where many different planning problems need to be solved, or the domain description evolves over time. For instance, in an application domain such as robotics, a mobile service robot may need to solve planning tasks such as collecting documents, making deliveries, or providing navigation assistance to visitors cambon2009hybrid (); erdem2012answer (); khandelwal2017bwibots (). It is convenient to achieve all these tasks using knowledge declared in a single description of the domain, and general purpose planning systems are well suited to the task.

In order to design a general purpose planning system, a language for formalizing the domain first needs to be selected, followed by the selection of a suitable solver which supports this language. Many different factors affect this selection process. Every language has its limitations in representing task planning problems, and given a particular language, specific language-dependent techniques may need to be employed to succinctly formalize a particular planning problem. A solver is also typically tied to a particular language, but may not support all features in that language, requiring careful construction of the domain description using only supported features. Additionally, the properties of the domain can affect how quickly a given pair of language and solver can solve planning problems. Finally, given a language, a solver and a planning problem, there can be many ways of formalizing the problem using the language. For these reasons, careful consideration needs to be given to the selection of language and solver.

This paper aims to help in the language selection process, given a task planning problem at hand. Specifically, we compare two declarative languages: the Planning Domain Definition Language (PDDLmcdermott1998pddl (), the most popular planning language that is used in the International Planning Competition (IPC), and Answer Set Programming (ASPgelfond2014knowledge (); lifschitz2008answer (), a popular general knowledge representation and reasoning (KRR) language that has been recently used in a variety of task planning problems lifschitz2002answer (); yang2014planning (); erdem2016applications (). PDDL was created for the explicit purpose of solving planning problems, whereas the development of ASP has focused on a broader set of reasoning tasks, such as inference and diagnosis, as well as planning.

The main contribution of this paper is a comparison of planning time between ASP-based and PDDL-based task planners when both are used to model the same domain. Evaluation is performed across three different benchmark problems. While it may be possible to construct ASP and PDDL planners specifically suited to these specific benchmarks, domain-independent solvers are compared in this paper. Although planner performance can be sensitive to domain encoding, we take care, to the extent possible, to encode the domains similarly in each language. The benchmark problems consist of the Blocks World and Hiking problems from the IPC coles2012survey (), as well as a variant of the Robot Navigation problem yang2014planning (). The Robot Navigation problem typically requires more complex reasoning about action preconditions and effects than problems in the IPC, which focus on generating long plans efficiently. Various properties of the domain or task in these benchmarks are also varied during evaluation to analyze the effect on planning time. The goal of this paper is to help a practitioner understand the effects of specific domain properties to aid the choice of language selection for general purpose planning.

We hypothesize that current state-of-the-art PDDL-based planners perform better on tasks with long solutions, and ASP-based planners tend to perform better on shorter tasks with a large number of objects. The hypothesis is confirmed in all three benchmark domains. We also hypothesize that ASP-based planners outperform PDDL-based planners in domains where complex reasoning is required to reason about action preconditions and effects. This is observed in a controlled experiment of the Robot Navigation and Blocks World domains. We specify what we mean by complex reasoning in Section 3.1.

2 Background

Research in task planning dates back to one of the earliest research areas in Artificial Intelligence. Since the development of STRIPS fikes1971strips (), many languages have been developed for representing task planning domains. Such languages typically need to describe actions’ preconditions and effects, and are commonly known as action languages. A summary of some early action languages is available gelfond1998action ().

In order to plan for real-world problems, action languages first need to be capable of formally representing complex planning domains. Some of the recent research in task planning is focused on developing languages that improve the representation capability of action languages giu04 (); lee13 (); babb2015action (). Planning using these action languages usually requires a translation to more general knowledge representation and reasoning (KRR) languages such as ASP. A planning paradigm for ASP was proposed lifschitz2002answer (), and has been used in many real-world applications chen2010developing (); erdem2012answer (); yang2014planning (). In this paper, we follow the same planning paradigm when encoding domains in ASP.

In parallel, another line of research in task planning aims at more efficient planning algorithms and their implementations. PDDL mcdermott1998pddl () was developed as a common formalism with the goal of allowing more direct comparison of planning algorithms and implementations. Since then, many efficient search algorithms have been developed for task planning problems, such as Fast-Forward hoffmann2001ff () and Fast-Downward helmert2006fast (). These algorithms have publicly available implementations including SAYPHI de2007using (), LAMA richter2011lama () and FDSS helmert2011fast ().

PDDL requires axioms, in the form of logical formulas, for reasoning within a situation (whereas action descriptions are used for reasoning across successive situations). A fundamental difference between PDDL and ASP is on their (non)monotonicity property. The axiom-based reasoning in PDDL is monotonic in the sense of logical reasoning, meaning that previously achieved conclusions remain when new information becomes available. In contrast, ASP is nonmonotonic, so it allows removal of previously achieved conclusions given new information. The nonmonotonic property of ASP makes it useful in tasks that require default reasoning and reasoning about inertial facts. Existing research has studied translating PDDL programs to ASP gebser2011plasp (), applying axioms extracted from PDDL programs to ASP-based planning ICAPS1715730 (), and switching between ASP-based and PDDL-based planning systems on mobile service robots lo2018petlon (). However, none of this research conducted empirical comparisons over the performances of the state-of-the-art PDDL-based and ASP-based planning systems.

3 Domain Formalization

In this section, we introduce the three benchmark domains, namely Robot Navigation, Blocks World and Hiking, and formally describe each in both ASP and PDDL.

It may be possible that different styles of encoding would result in different planning times. To ensure the conclusion is general and fair, we select the benchmark domains from a variety of origins, and follow the encoding in existing literatures. Blocks World and Hiking have been used in the IPC, and PDDL-based planners have been specifically designed to solve problems in the IPC. The PDDL versions are used as is, and translated into equivalent ASP code. Robot Navigation is a domain used to demonstrate planning using an ASP description yang2014planning (), and contains properties such as recursive action effects that are missing from IPC domains. We translate the complex reasoning rules of the Robot Navigation domain from ASP as PDDL axioms.

Though the languages themselves require certain differences in domain encodings, and direct comparisons are complicated by the fact that typically the people doing the encoding have greater familiarity with one language or the other, we ensure the fairest comparison possible by enforcing in all translations that both the ASP and PDDL versions for a given domain have exactly the same set of predicates and actions, along with the same preconditions and effects.2 Specifically, an action should be allowed to execute on the same set of states, and it should make the same change to the state, regardless of the language. Consequently, as we use optimal ASP and PDDL planners in the experiments, they generate identical plans, and only planning times need to be compared. A detailed explanation of how ASP can be used for planning is available in previous work lifschitz2002answer ().

3.1 Robot Navigation

The robot navigation domain differs from classical planning domains in the IPC in that complex reasoning needs to be performed to ensure that action preconditions are met, and that action effects are executed correctly. Specifically, this domain features action effects that require recursive reasoning to compute the final state of the world. In this domain, a mobile robot navigates an office floor which consists of a set of rooms that are connected to one another. Rooms can be connected to one another via doors, and closed doors need to be opened by the robot before it can pass through. Alternatively, rooms can be directly connected to one another such that access is always possible from any location in one room to any location in the other.

The robot has the following perception and actuation modules available. Using a low-level controller, the robot can traverse to any room from its current location if its path is not blocked by a closed door, and this navigation can be encoded by a single high-level symbolic action goto. Furthermore, the robot has some means of opening a closed door when it is next to it, either by enlisting human aid, or using a robot arm to open the door. On the perception side, the robot can sense its location, whether or not it is next to a door, and whether or not a door is open.

The domain knowledge can be formalized in ASP by statements defined using the following predicates:

  • hasdoor(R,D): This predicate specifies that room R has door D to move to an adjacent location. Statements expressed using hasdoor are specified during initialization, and do not change over time. The PDDL expression is (hasdoor ?r - room ?d - door).

  • connected(R1,R2): The connected predicate indicates that room R1 is directly connected to room R2 without a door. Similar to hasdoor, this predicate is used during initialization to describe directly connected locations. In PDDL, statements are described as (connected ?r1 - room ?r2 - room).

  • acc(R1,R2): acc specifies that room R1 is accessible from room R2 via a single navigation action executed by a low level controller. Intuitively, any two rooms that are not separated by a closed door are accessible, i.e. the low level controller can navigate from one room to another.

    R1 is accessible from R2 if R1 is directly connected to R2, or R1 and R2 share the same door D which is open. Furthermore, acc is both commutative, i.e. R1 is accessible from room R2 if R2 is accessible from R1, as well as associative, i.e. if both R1 and R2 are accessible from r3, then they’re accessible from one another. This associative property requires a recursive definition:

    {Verbatim}

    [fontfamily=courier, fontseries=b] acc(R1,R2,n) :- connected(R1,R2). acc(R1,R2,n) :- open(D,n), hasdoor(R1,D), hasdoor(R2,D). acc(R1,R2,n) :- acc(R2,R1,n). acc(R1,R2,n) :- acc(R1,R3,n), acc(R3,R2,n).

    The recursive formulation of acc can be expressed in PDDL using derived predicates:

    {Verbatim}

    [fontfamily=courier, fontseries=b] (:derived (acc ?r1 - room ?r2 - room) (or (connected ?r1 ?r2) (exists (?d - door) (and (open ?d) (hasdoor ?r1 ?d) (hasdoor ?r2 ?d))) (acc ?r2 ?r1) (exists (?r3 - room) (and (acc ?r1 ?r3) (acc ?r3 ?r2)))))

  • at(R,n): at is used to specify that the robot is at room R at timestep n (of the high-level plan). This predicate is inertial, i.e. the robot remains in room R if there is no evidence showing it is not in room R anymore, and this property is specified as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] at(R,n) :- at(R,n-1), not -at(R,n).

    In PDDL, the predicate is expressed as (at ?r - room), and all predicates are inertial by default.

  • open(D,n): Door D is open at step n. door is inertial, i.e. the robot believes that a door will stay in the same state unless sensed differently. In ASP, the inertial property for this predicate is represented as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] open(D,n) :- open(D,n-1), not -open(D,n).

    In PDDL, open is expressed as (open ?d - door).

  • canopen(D,n): The robot can open door D if it is right next to it. canopen is the action effect of approaching a door, and a precondition before the door can be opened. canopen is not inertial. In PDDL, the predicate is expressed as (canopen ?d - door).

  • visited(R,n): Once the robot visits a room, the visited fluent for that room remains true until the end of planning process. visited is used to describe goal conditions. The persistence property is expressed in ASP as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] visited(R,n) :- visited(R,n-1).

    (visited ?r - room) expresses visited in PDDL.

There are three actions in the domain: goto, approach, and opendoor, with the following definitions:

  • goto(R2,n): This action specifies that the robot should navigate to room R2 at timestep n in the high-level plan. The precondition for this action is that robot must be in a room R1 from which R2 is accessible. Once the robot goes to room R2, the goal condition visited is set to true for that room as well. The ASP rules defining the action preconditions and effects are as follows:

    {Verbatim}

    [fontfamily=courier, fontseries=b] :- goto(R2,n), at(R1,n-1), not acc(R1,R2,n-1). at(R2,n) :- goto(R2,n). -at(R1,n) :- goto(R2,n), at(R1,n-1), R1 != R2. visited(R2,n) :- goto(R2,n).

    The same description in PDDL is expressed as follows:

    {Verbatim}

    [fontfamily=courier, fontseries=b] (:action goto :parameters (?r2 - room) :precondition (exists (?r1 - room) (and (at ?r1) (acc ?r1 ?r2))) :effect (and (at ?r2) (forall (?r1 - room) (when (at ?r1) (not (at ?r1)))) (visited ?r2) (forall (?d1 - door) (not (canopen ?d1)))))

    The action effects in PDDL have an additional statement than ASP to indicate that canopen is not inertial.

  • approach(D,n): This action specifies that the robot should approach door D. The action is only executable when the robot is in a room that has door D. After executing this action, the robot can open door D. In ASP, this action is expressed as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] :- approach(D,n), at(R1,n-1), not hasdoor(R1,D). canopen(D,n) :- approach(D,n).

    In PDDL, this action is expressed as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] (:action approach :parameters (?d - door) :precondition (exists (?r1 - room) (and (at ?r1) (hasdoor ?r1 ?d))) :effect (and (canopen ?d) (forall (?d1 - door) (when (not (= ?d1 ?d)) (not (canopen ?d1))))))

    The action effects in PDDL both express that door d can be opened, and no other doors in the domain can be opened without approaching them first.

  • opendoor(D,n): This action allows the robot to open door D if canopen(D) is true. Opening an open door does not change the state of the world. This action is represented in ASP as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] :- opendoor(D,n), not canopen(D,n-1). open(D,n) :- opendoor(D,n).

    In PDDL, this action is represented as:

    {Verbatim}

    [fontfamily=courier, fontseries=b] (:action opendoor :parameters (?d - door) :precondition (canopen ?d) :effect (and (open ?d) (forall (?d1 - door) (not (canopen ?d1)))))

In all three actions, a specific action effect in PDDL describes the non-inertial property of canopen. This effect is not required in ASP because the canopen is not inertial.

The goal in the Robot Navigation domain is to visit a randomly selected set of rooms. In order to visit a room, the robot needs to recursively reason about which rooms are accessible from one another. If a door in its path is closed, the robot needs to explicitly approach the door and execute an opendoor action. Whenever opendoor is executed, the direct accessibility of rooms changes, and this action effect needs to be computed recursively. This recursive property of the domain differentiates it from traditional IPC planning domains such as Blocks World and Hiking.

3.2 Blocks World

In the Blocks World domain, the goal is to move a set of stackable blocks from one configuration to another using a robot hand. We use the official domain definition in IPC-2011 as the PDDL domain description, and translate the domain to ASP. The full description of the IPC domain is available online in PDDL3 and ASP4. Due to space constraints, we only describe the translation of the pick-up action to ASP, and the other actions (put-down, stack, and unstack) follow similarly. pick-up allows a robot to pickup a block that has no blocks underneath it (designated by ontable). Furthermore, it should also have no blocks stacked on top of it (designated by clear). Finally, a block can only be picked up if the robot hand is empty (designated by handempty). The effect of the action is that the robot hand is holding the block, and all preconditions for the action become false. This action description translated to ASP looks as follows:

{Verbatim}

[fontfamily=courier, fontseries=b] :- pickup(B,n), not clear(B,n-1). :- pickup(B,n), not ontable(B,n-1). :- pickup(B,n), not handempty(n-1). -ontable(B,n) :- pickup(B,n). -clear(B,n) :- pickup(B,n). -handempty(n) :- pickup(B,n). holding(B,n) :- pickup(B,n).

The first three statements specify the same action preconditions specified in the PDDL description, and the last four statements specify the same action effects as specified in the PDDL description.

The extended version of the Blocks World domain introduces a recursively defined predicate: above THIEBAUX200538 (). The PDDL definition is as follows:

{Verbatim}

[fontfamily=courier, fontseries=b] (:derived (above ?x ?y) (or (on ?x ?y) (exists (?z) (and (on ?x ?z) (above ?z ?y)))))

The predicate is defined in ASP as:

{Verbatim}

[fontfamily=courier, fontseries=b] above(X,Y,n) :- on(X,Y,n). above(X,Y,n) :- on(X,Z,n), above(Z,Y,n).

The complete domain description in PDDL5 and ASP6 are online. We use both the original and the extended versions of the Blocks World domain in experiments.

3.3 Hiking

We select the hiking domain, new in IPC-2014, as our third benchmark domain. The hiking domain features negative preconditions. We use the official PDDL formalization in IPC, and an equivalent definition in ASP for this study.

In short, the purpose of this domain is to arrange activities for a number of couples so each couple can hike to their destination with a tent ready. A hiking problem specifies connections between places, and initial locations of couples, cars, and tents. A typical plan transports and sets up tents at the destination, drives each couple to the starting point of their hike, and then has them walk together along the hike. The complete description of the Hiking domain is available online for both PDDL7 and ASP8.

4 Experiments

The experiments in this section are designed to compare planning times when the domains and problems formalized in the previous section are optimally solved using state-of-the-art solvers. Our encoding strategy and the award-winning optimal planning systems ensure that all planners generate the same plan given the same pair of domain and problem.

Since IPC 2014 does not take into account or announce the time needed to solve each problem, most PDDL planners always use the maximum time allowed, and are therefore unfit for a comparison of planning times. We select FastDownward helmert2006fast () with the setting FDSS-1 helmert2011fast (), which had the highest score in the sequential optimization track at IPC 2011. The Robot Navigation domain and the extended Blocks World domain require derived predicates. Since none of the planners in the optimization track support derived predicates, we use FastDownward with the setting LAMA-2011 richter2011lama (), the winner of sequential satisficing track in 2011. We use version 4.5.4 of Clingo GebserKKS14 () in incremental mode as the ASP solver. Clingo is an Answer Set solving system that integrates Clasp, the winner of the fifth Answer Set Programming Competition in 2015 calimeri2016design ().

Planner performances are evaluated on a general-purpose High Throughput Computing (HTC) cluster that is operated by the Department of Computer Science at UT Austin. We filter out machines with less than 8GB memory in the experiments, resulting in more than ten machines with different hardware configurations (e.g., memory ranging from 8GB to more than 500GB). All data points are averaged across at least 10 trials to reduce noise, and standard deviations are reported. Given the wide range of hardware configurations and statistical analysis, we aim to conclude with observations that are generally valid and useful to practitioners.

During evaluations, various domain characteristics are also changed to measure the difference in performance of different planning paradigms. It should be noted that we are generally more interested in comparing sensitivities of planning systems (instead of comparing individual data points) given different domain characteristics. The sensitivity that can be reflected by the “trend” of a series of data points is typically more robust to implementation details of planning systems such as programming languages and compilers. The goal of these evaluations is to test the hypothesis that PDDL-based approaches work better in situations where the generated plans have many steps, and ASP-based approaches work better in situations where the domain is large or substantial reasoning is required at every step of the plan to compute the world state.

Figure 1: Robot Navigation: Small Domain - 10 rooms, Large Domain - 15 rooms. 40% of rooms are connected via doors, and the rest are directly accessible from the corridor.

Robot Navigation

Results from the Robot Navigation domain are shown in Figure 1. Two different versions of the domain have been created. The small domain has 10 rooms, and each of them is connected a central corridor. 4 rooms are connected via doors, and the rest are directly connected. In contrast, the large domain has 15 rooms where 6 rooms are connected via doors to the corridor.

The robot is initially located in the corridor, and as its goal, needs to visit a number of rooms (represented on the x-axis in Figure 1) that are randomly selected in each trial. Since rooms that are connected by doors take more steps to visit, we increase the number of trials (to 50), on which we compute the average planning time for each data point. Regardless, the number of rooms in the goal is positively correlated with the plan length. The planning time for each planner is represented on the y-axis (log scale, same for all following figures).

We can observe that the red curves have smaller slopes, since they intersect (or will do so) with the blue curves. This confirms our hypothesis that PDDL planners are better at solving planning problems which require a large number of steps. We can also observe that the gap between red curves is larger than the gap between blue curves, even on logarithmic scale. This observation again supports the hypothesis that ASP-based planning is less sensitive to object scaling.

Furthermore, ASP-based planning is much faster than PDDL-based planning when the number of rooms is less than eight, and finishes within a reasonable amount of time. This is especially useful in domains such as robotics where fast real-time operation is necessary. This better performance is probably a consequence of recursive action effects embedded in the domain. We further verify this observation in a controlled experiment using the Blocks World domain.

Figure 2: Blocks World: Small Domain - 15 blocks, Large Domain - 60 blocks. We use a timeout of 1800 seconds (same as IPC). The graph only plots configurations where all trials of both planners finished before timeout (same for all following experiments).
Figure 3: Blocks World: Regular Domain vs. Extended Domain.
Figure 4: Varying individual domain attributes in the Hiking domain
Figure 5: Hiking: Small Domain vs. Large Domain

Blocks World

Figure 2 reports results from the regular Blocks World domain. Similarly, two versions of the domain are evaluated: a small domain with 15 blocks in the environment, and a large domain with 60 blocks in the environment. Initially, all blocks are unstacked and on the table (so as to control the optimal plan length). The goal of the planners is to generate a plan that builds a single stack of a specified height with randomly specified blocks. The plan length is proportional to the height of the stack, and is represented as the x-axis in Figure 2. All results have been averaged across 10 trials.

The two red lines are far apart and almost horizontal, i.e.  the PDDL planner slows down significantly in the larger domain, but planning time does not depend on plan length. The planning time of Clingo grows as the plan length increases, but the difference between different sized domains is smaller than PDDL. These observations support our hypotheses that PDDL-based planning is fast at producing long plans, while ASP-based planning is faster in large domains where smaller plans are necessary.

Figure 3 compares the performance of each planner between the regular Blocks World domain and the extended version. In this experiment, the domain has 10 blocks. The task is the same as before: building a tower of various heights from unstacked blocks. The difference of the extended domain is that the goal condition is expressed with the predicate above, where the planners reason about recursive effects. Similarly, the results are averaged across 10 trials.

In Figure 3, there is a small difference in ASP solving time, but a significant change in the performance of the PDDL planner. In contrast to Figure 2, we can observe that complex reasoning affects PDDL planning time in its slope with respect to plan length, whereas object scaling shifts the curve up. The observation confirms that ASP-based planning is better for domains that involve complex reasoning (reasoning about recursive action effects in this case).

Hiking

Figure 4 and 5 show results from the Hiking domain. In each graph, the x-axis is the total number of trips made by all couples, and the y-axis is the planning time in seconds. Figure 5 shows the performance of each planner for two sizes of the domain. Although we can make similar observations that the PDDL planner becomes faster than the ASP planner at higher plan lengths but slower with more objects, the evidence is weaker than the results above. Since the Hiking domain has four types of objects that affect planning in different ways, the domain size has four dimensions. Figure 4 shows a more controlled set of experiments in which the larger domain increases only one type of objects. Objects are added in a way that does not affect the plan length or the number of optimal plans at each point of x-axis. In all graphs the slope of blue curves is larger than the slope of red curves. So the hypothesis about the plan length holds for all object types.

When adding cars and tents to the domain, from Figure 4(a) and 4(b), we can observe that the ASP solver is less sensitive than the PDDL planner, but there is no significant difference in the case of couples and places – see Figure 4(c) and 4(d). We find that couples and places are more heavily used as action parameters than other two types of objects. Actions such as drive_passenger and walk_together even take two parameters of each. Therefore, increasing the number of these objects complicates the grounding9 of ASP problems. We also observe that most of the increased planning time of Clingo is in grounding. For instance, when the number of couple increases from 5 to 10 – see Figure 4(c), average solving time for one trip stays below 0.8 second, while average grounding time grows from 16.0 seconds to 354.2 seconds. Based on these two observations, we hypothesize that if the domain has actions that check or change the state of many objects, Clingo’s advantage at planning in large domains can be canceled out by the extra grounding time. A further analysis of how parameter type and number affect grounding time is left for future research.

Remark

All of the above results support our hypothesis that PDDL-based approaches work better in situations where the generated plans have many steps, and ASP-based approaches work better in situations where the domain is large or substantial reasoning is required at every step of the plan to compute the world state.

Given the wide range of hardware configurations of machines in the HTC cluster and the low standard deviation values reported in the results, we can see machines of different configurations did not cause significant differences in planning time, and hence the trends are generally consistent across the different machine types.

5 Conclusions and Discussions

In this paper, we evaluated ASP-based and PDDL-based planners using three benchmark domains encoded as similarly as possible. The analysis in this paper demonstrates that PDDL-based planners perform better when tasks require long solutions. However, ASP-based task planners are less susceptible to an increase in the number of domain objects, as long as the growth in the number of objects does not explode the number of grounded actions. Finally, in domains requiring complex reasoning such as the Robot Navigation domain, ASP-based planners can be considerably faster than PDDL-based planners for shorter plans.

This paper, by no means, aims to provide a list of the best planners given a pair of planning domain and planning problem, which is infeasible in practice. From a high-level perspective, the observations shared in this paper are intended to contribute to the community’s understanding of how properties of planning domains and problems affect the performance of different planning systems. Based on one’s knowledge and intuition about properties of the planning problems at hand, a practitioner can then make a more informed choice of the planning system.

In this work, we selected the Clingo system for evaluating the ASP-based planning formalism, and the FastDownward planning systems of FDSS-1 and LAMA-2011 for the PDDL-based formalism. These award-winning optimal systems represent the state-of-the-art algorithms and implementations of the two planning paradigms. We did not include more planning systems into the experiments and leave that as future work, because comparisons using less effective systems could be problematic – performance could be degraded by deficiencies in the planners themselves.

As the first work on empirical comparisons of PDDL-based and ASP-based planners, we focus on a clear presentation of the methodology and results, so we limit the quantity of experiments to a range that can be concisely analyzed in detail. We carefully selected the three domains that are as distinct as possible for a representative comparison. We believe the conclusions hold in most cases, and we leave further evaluations over the two planning formalisms and their implementations for future work.

Footnotes

  1. thanks: Jiang and Stone are with the Department of Computer Science, The University of Texas at Austin. Zhang is with the Department of Electrical Engineering and Computer Science, Cleveland State University. Khandelwal is with Cogitai, Inc. This work was conducted when Zhang and Khandelwal were with The University of Texas at Austin.
  2. Though we acknowledge that different encodings may be more suitable to each planner, we know of no clear way to control for that.
  3. http://pastebin.com/raw/b07aMTJB
  4. http://pastebin.com/raw/SAqM3xbF
  5. http://pastebin.com/raw/FwZgGmZf
  6. http://pastebin.com/raw/9D2PuNze
  7. http://pastebin.com/raw/v3wkv57W
  8. http://pastebin.com/raw/Dw1BwG0Z
  9. Plan generation in ASP is a two-step process that includes grounding and solving, where the grounding step outputs a variable-free representation.

References

  1. Babb, J., Lee, J.: Action language BC+: preliminary report. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), pp. 1424–1430. AAAI Press (2015)
  2. Calimeri, F., Gebser, M., Maratea, M., Ricca, F.: Design and results of the fifth answer set programming competition. Artificial Intelligence 231, 151–181 (2016)
  3. Cambon, S., Alami, R., Gravot, F.: A hybrid approach to intricate motion, manipulation and task planning. The International Journal of Robotics Research 28(1), 104–126 (2009)
  4. Chen, X., Ji, J., Jiang, J., Jin, G., Wang, F., Xie, J.: Developing high-level cognitive functions for service robots. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pp. 989–996. International Foundation for Autonomous Agents and Multiagent Systems (2010)
  5. Coles, A., Coles, A., Olaya, A.G., Jiménez, S., López, C.L., Sanner, S., Yoon, S.: A survey of the seventh international planning competition. AI Magazine 33(1), 83–88 (2012)
  6. Erdem, E., Aker, E., Patoglu, V.: Answer set programming for collaborative housekeeping robotics: representation, reasoning, and execution. Intelligent Service Robotics 5(4), 275–291 (2012)
  7. Erdem, E., Gelfond, M., Leone, N.: Applications of answer set programming. AI Magazine 37(3) (2016)
  8. Fikes, R.E., Nilsson, N.J.: Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence 2(3-4), 189–208 (1971)
  9. Gebser, M., Kaminski, R., Kaufmann, B., Schaub, T.: Clingo = ASP + control: Preliminary report. CoRR abs/1405.3694 (2014). URL http://arxiv.org/abs/1405.3694
  10. Gebser, M., Kaminski, R., Knecht, M., Schaub, T.: plasp: A prototype for PDDL-based planning in ASP. Logic Programming and Nonmonotonic Reasoning pp. 358–363 (2011)
  11. Gelfond, M., Kahl, Y.: Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach. Cambridge University Press (2014)
  12. Gelfond, M., Lifschitz, V.: Action languages (1998)
  13. Giunchiglia, E., Lee, J., Lifschitz, V., McCain, N., Turner, H.: Nonmonotonic causal theories. Artificial Intelligence 153(1-2), 49–104 (2004)
  14. Helmert, M.: The fast downward planning system. Journal of Artificial Intelligent Research (JAIR) 26, 191–246 (2006)
  15. Helmert, M., Röger, G., Karpas, E.: Fast downward stone soup: A baseline for building planner portfolios. In: ICAPS 2011 Workshop on Planning and Learning, pp. 28–35. Citeseer (2011)
  16. Hoffmann, J.: FF: The fast-forward planning system. AI magazine 22(3), 57 (2001)
  17. Khandelwal, P., Zhang, S., Sinapov, J., Leonetti, M., Thomason, J., Yang, F., Gori, I., Svetlik, M., Khante, P., Lifschitz, V., Aggarwal, J.K., Mooney, R., Stone, P.: BWIBots: A platform for bridging the gap between AI and human–robot interaction research. The International Journal of Robotics Research 36(5-7), 635–659 (2017)
  18. Lee, J., Lifschitz, V., Yang, F.: Action language BC: preliminary report. In: Proceedings of the Twenty-Third international joint conference on Artificial Intelligence (IJCAI), pp. 983–989. AAAI Press (2013)
  19. Lifschitz, V.: Answer set programming and plan generation. Artificial Intelligence 138(1), 39–54 (2002)
  20. Lifschitz, V.: What is answer set programming? In: Proceedings of the 23rd national conference on Artificial intelligence, pp. 1594–1597. AAAI Press (2008)
  21. Lo, S.Y., Zhang, S., Stone, P.: PETLON: Planning efficiently for task-level-optimal navigation. In: Proceedings of the 17th Conference on Autonomous Agents and MultiAgent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (2018)
  22. McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D.: Pddl-the planning domain definition language (1998)
  23. Miura, S., Fukunaga, A.: Automatic extraction of axioms for planning. In: Proceedings of the Twenty-Seventh International Conference on Automated Planning and Scheduling (ICAPS), pp. 218–227 (2017)
  24. Richter, S., Westphal, M., Helmert, M.: Lama 2008 and 2011. In: International Planning Competition, pp. 117–124 (2011)
  25. De la Rosa, T., Olaya, A.G., Borrajo, D.: Using cases utility for heuristic planning improvement. In: International Conference on Case-Based Reasoning, pp. 137–148 (2007)
  26. Thiébaux, S., Hoffmann, J., Nebel, B.: In defense of PDDL axioms. Artificial Intelligence 168(1-2), 38–69 (2005)
  27. Yang, F., Khandelwal, P., Leonetti, M., Stone, P.: Planning in answer set programming while learning action costs for mobile robots. In: 2014 AAAI Spring Symposium Series (2014)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
167617
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description