An Analytics Tool for Exploring Scientific Software and Related Publications

An Analytics Tool for Exploring Scientific Software and Related Publications

Abstract

Scientific software is one of the key elements for reproducible research. However, classic publications and related scientific software are typically not (sufficiently) linked, and it lacks tools to jointly explore these artefacts. In this paper, we report on our work on developing an analytics tool for jointly exploring software and publications. The presented prototype, a concept for automatic code discovery, and two use cases demonstrate the feasibility and usefulness of the proposal.

Keywords:
software reproducibility source code exploration cross-modal relations.

1 Introduction

The open science movement works towards the general availability of scientific insight and is considered one answer to the so-called ”reproducibility crisis” [2]. Science results are often generated by a combination of software, data, and parameter settings, all of which contribute to the final result (and its interpretation). The complexity of all these elements is hardly describable in a single article – and often the publication does not allow the full reproduction of the achieved results. In the line of work towards consequent reproducibility of scientific results, there are three main tasks to be tackled: (a) motivate researchers to reproduce past results; (b) develop novel ways for the integrated presentation of future scientific results; (c) develop tools which allow for easy exploration of existing scientific works.

The work at hand focusses on the two latter objectives. It presents a tool which facilitates the examination of existing research involving software by joint exploration of scientific article and respective source code. The prototype allows the exploration of both in one interface, and the semi-automatic creation of semantic relations between them. The software is extended by basic visualisations.

This kind of work is related to research areas, which have been active for decades: (a) automatic code analysis, and (b) the automatic analysis of scientific publications. Solutions for automatic code analysis aim at generating textual documentation (e.g. [7]), summarising code (e.g. [8]), or at generating visualisations (e.g. [4]). Also common is the generation of formal code models using semantic technologies (e.g. [1]) or logical constructs , as realised in tools such as JTransformer1.

While there is much work on linking code to other (textual) resources (e.g. traceability [3]), to documentation (e.g. [4]), on the automatic understanding of scientific publications (e.g. [5]), or on linking publications with software and archiving them [6], there has been little work on joint analytics of scientific software and publications yet (e.g. [9]).

2 SciSoftX: Scientific Software Explorer

The Software Explorer provides researchers with functionalities for the exploration of external article-software ensembles and/or annotation of own works for better comprehensibility. Its final version will provide functionalities such as (a) manual annotation of article-software relations; (b) semi-automatic discovery of relations; (c) visualisations for relation exploration.

Figure 1: Main window of the GUI: Linked code references are highlighted in colour.

2.1 Functionality

SciSoftX allows the user to open and simultaneously view a software project and a publication (Fig. 1). Parsing and processing of source code is realised using ANTLR2 (Another tool for Language Recognition) that supports most of the relevant programming languages, while publications are processed via PDF.js3. The user can manually link code identifiers to relevant locations in the publication. When the user moves the mouse over a linked identifier in the publications, a tool tip shows the relevant source code positions.

Automatic Discovery of Code Identifiers and Snippets: At the current stage, the tool contains a very basic method for code-relevant text snippets: It relies on the common convention of setting code elements in monospace fonts. The found identifiers are used to search the code model produced by ANTLR, multiple finds are disambiguated based on vicinity. In a random sample of 24 articles from computer science, the monospace-based linker was able to correctly detect 89,9% of the links annotated by a human expert.

Manual Annotation of Links: As a facilitator of exchange between scientists the tool also allows for the manual annotation of resources. In a step-wise process, the user marks article snippets, code elements and types the established link with one of the pre-defined labels. The created set of links can be exported to an xml format and imported by an interested reader.

Figure 2: Graph-based views on connections between software and publication. Red nodes: Mentions in publications. Blue nodes: Source code files (left) or packages (right).

Visualisations: Graph-based visualisations illustrate relations between software and publication on different levels of abstraction. Fig. 2 (left) depicts connections between source code and class files with exact positions in the publication (that is, page and line number); Fig. 2 (right) displays connections at the package (software component) and page (publication) level.

2.2 Use cases

Use case 1 – Reader-side: A researcher reads a publications that refers to a blob of software. She/he tries to understand the structure and rationale of the software. This time-consuming task can be supported by the automatic creation of links between textual description and actual source code, and the visualisations provided by SciSoftX. The user can click on nodes in the visualisation or on text elements that are highlighted in the publication and explore the implementation details, discover additional parameters, and understand the relevant code part step by step. Furthermore, she/he can manually add and save useful information and metadata, which can help future users to explore the software.

Use case 2 – Author-side: A paper author wishes that his/her software can be easily understood, e.g. in a reviewing process or for re-use. Therefore, she/he uses the SciSoftX’s manual and automatic methods to annotate the semantic relations between his/her paper and the underlying software and publishes the annotations. The visualisation of cross-modal relations can aid the author (and the reviewer) to decide whether all relevant code parts and parameters are covered by the publication. In this way, the tool helps to evaluate the quality of the software description in a paper.

3 Conclusion

Reproducibility is one of the major issues of today’s scientific landscape. In this paper, we have reported on work in progress for an analytics tool that allows users to explore relations between scientific software and publications. To this date, the tool features simple mechanisms for detecting links between software and publications who serve as a proof of concept.

Future works will explore (a) more powerful infrastructures for code analysis ; (b) more sophisticated means for text/image analysis, e.g. mapping diagrams and formulas to source code.

Footnotes

  1. http://sewiki.iai.uni-bonn.de/research/jtransformer/start
  2. http://www.antlr.org/
  3. https://mozilla.github.io/pdf.js/

References

  1. Atzeni, M., Atzori, M.: Codeontology: Rdf-ization of source code. In: Proc. of the ISWC 2017, Vienna, Austria, October 21-25, 2017. LNCS, vol. 10588, pp. 20–28. Springer (2017). https://doi.org/10.1007/978-3-319-68204-4_2
  2. Baker, M.: 1,500 scientists lift the lid on reproducibility. Nature News 533(7604),  452 (2016)
  3. Borg, M., Runeson, P., Ardö, A.: Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empirical Software Engineering 19(6), 1565–1616 (2014). https://doi.org/10.1007/s10664-013-9255-y
  4. Chen, X., Hosking, J.G., Grundy, J.: Visualizing traceability links between source code and documentation. In: IEEE Symposium on Visual Languages and Human-Centric Computing, Innsbruck, Austria. pp. 119–126 (2012). https://doi.org/10.1109/VLHCC.2012.6344496
  5. Constantin, A.: Automatic structure and keyphrase analysis of scientific publications. Ph.D. thesis, University of Manchester, UK (2014), http://www.manchester.ac.uk/escholar/uk-ac-man-scw:230124
  6. Holzmann, H., Sperber, W., Runnwerth, M.: Archiving software surrogates on the web for future reference. In: Intl Conf. on Theory and Practice of Digital Libraries. pp. 215–226. Springer (2016). https://doi.org/10.1007/978-3-319-43997-6_17
  7. Moser, M., Pichler, J.: Documentation generation from annotated source code of scientific software: position paper. In: Proceedings of the Int’l Workshop on Software Engineering for Science, SE4Science@ICSE 2016, Austin, Texas, USA, May 14-22, 2016. pp. 12–15. ACM (2016). https://doi.org/10.1145/2897676.2897679
  8. Nazar, N., Hu, Y., Jiang, H.: Summarizing software artifacts: A literature review. J. Comp. Sci. Tech. 31(5), 883–909 (2016). https://doi.org/10.1007/s11390-016-1671-1
  9. Witte, R., Li, Q., Zhang, Y., Rilling, J.: Text mining and software engineering: an integrated source code and document analysis approach. IET Software 2(1), 3–16 (2008). https://doi.org/10.1049/iet-sen:20070110
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
204726
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description