COINS: ContinuOus IntegratioN in wirelesS technology development
Network testing plays an important role in the iterative process of developing new communication protocols and algorithms. However, test environments have to keep up with the evolution of technology and require continuous update and redesign. In this paper, we propose COINS, a framework that can be used by wireless technology developers to enable continuous integration (CI) practices in their testbed infrastructure. As a proof-of-concept, we provide a reference architecture and implementation of COINS for controlled testing of multi-technology 5G Machine Type Communication (MTC) networks. The implementation upgrades an existing wireless experimentation testbed with new software and hardware functionalities. It blends web service technology and operating system virtualization technologies with emerging Internet of Things technologies enabling CI for wireless networks. Moreover, we also extend an existing qualitative methodology for comparing similar frameworks and identify and discuss open challenges for wider use of CI practices in wireless technology development.
continuous integration, wireless networks, software development, testbed, machine type communication
The next generation of mobile networks will have to support industrial and traffic safety applications, enhanced multimedia applications on smartphones and a plethora of connected meters, sensors and other devices that will comprise living and working environments in the future. In order to meet the requirements of all these applications, mobile networks will have to accommodate both human-type communications (HTC) and machine-type communications (MTC) . MTC capabilities of existing predominantly HTC-oriented cellular technologies will be enhanced through the inclusion of non-cellular short-range and low-power wide area technologies under the 5G umbrella in the form of capillary networks. This will result in a multi-technology, high-interference radio environment  such as can be found in some frequency bands in dense urban centers. In Figure 1, spectrum measurements in the shared 200 kHz unlicensed sub-1 GHz band over a course of 24 hours show signals from IEEE 802.15.4-based, LoRa and Sigfox networks as well as a number of unidentifiable proprietary technologies.
Given the increasing complexity of wireless technologies, evaluation and testing of new devices, protocols or solutions in realistic environments is desirable but tedious and expensive. Although complicated and expensive to build and maintain, distributed and heterogeneous wireless testbeds offer the most realistic experimental conditions . While the cost of adding additional hardware functionality to support emerging technologies is difficult to overcome, the cost of developing software for the increased variety and number of wireless devices can be significantly reduced using continuous integration (CI) principles and tools . In particular, open source software projects have benefited significantly from adopting CI principles that significantly reduce the workload of the core maintainers. Contributed code can be automatically tested for syntax, style and errors using sophisticated toolchains that automate previously manual development practices. Studies have shown that CI methodology significantly increases productivity and code quality .
A recently proposed software-centric systematic testing methodology for mobile networks  incorporates unit testing, integration testing and system testing, which are central to software development and represent an important part of the CI process. Testbed support enabling such a methodology can be economically implemented using modern tools, such as software containers, that can be automatically deployed to test new code.
The main contribution of this paper is COINS, a framework for ContinuOus IntegratioN in wirelesS technology development that can be used by developers to enable conventional CI practices on wireless infrastructure. To illustrate and facilitate discussion on the framework capabilities, we also provide a reference architecture and implementation. In particular, the existing LOG-a-TEC testbed [7, 8] for controlled testing of emerging multi-technology 5G capillary MTC networks was upgraded to enable CI. LOG-a-TEC is a combined outdoor and indoor heterogeneous wireless testbed for experimentation with sensor networks and machine-type communication
Two further contributions include (i) extending an existing qualitative methodology  for comparing similar frameworks; and (ii) identification of and discussion on open challenges for wider adoption of CI practices in the development of wireless technologies.
The paper is structured as follows. Section 2 identifies the main functionalities of a wireless experimentation testbed required by a software developer and explains how the CI process can be utilized in handling them. Section 3 details the reference architecture and implementation of the COINS. Section 4 discusses the deployment of the new wireless experimentation technology as an extension of the existing LOG-a-TEC testbed and lists example supported experiments, while Section 5 compares COINS with related work. Section 6 identifies and discusses open challenges. Finally, Section 7 summarizes the paper.
2 Continuous integration process on wireless testbeds
There are four main functionalities that wireless network developers and testers need from a testbed environment:
Simple, low-effort integration of software and hardware components that require testing; the priority is to minimize the overhead that testbed usage introduces into the technology development cycle.
Controllable generation of traffic in the network under test, such as on-demand packet transmission according to custom patterns and distributions, to test new network protocols under realistic traffic conditions.
Controlled emulation of a radio operating environment using various technologies in order to identify potential coexistence issues between new and legacy technologies as early in the development cycle as possible.
Monitoring and analysis of network and radio spectrum metrics.
The introduction of the CI process into the wireless technology development proposed in this paper significantly lowers the barriers to the first and last of these functionalities while relying on the other two to successfully carry out the CI tests. According to existing CI practices, the proposed process should properly interface three main entities depicted in Figure 2a:
Repository that holds the code, configuration files, tests and documentation, i.e., everything needed for the automated build. Generally, the repository is implemented using a version control system such as Git or Mercurial.
Automation system that performs all the instantiations and builds and transfers software components. Conceptually, it does for CI what the production line does for a factory. Its implementation typically includes a large set of tools and components. In modern software development, the most efficient approach is to use web and containerization technologies.
Testbed device is used to run the tests and experiments. It can be off-the-shelf equipment, a custom built device or a hybrid. The first approach is more suitable for improving existing technologies, while the second is for developing new ones. For the sake of generality, we assume in the following that a testbed device comprises separate functional blocks represented by the infrastructure node for device management and the target node for test execution.
As depicted in Figure 2a, the CI process begins in Step 1 from when a new testbed device automatically registers itself in the automation system and becomes available for testing. In Step 2, the developer sends new code (pushes a commit) to the repository, initiating Step 3, which consists of triggering the automation system. In Step 4, the automation system pulls the changes from the repository and executes the deployment scripts specifying the testbed devices required by the test case. Step 5 deploys the changes from the repository to the testbed device and instructs it to start the build process. In Step 6, the testbed device initiates the build process on the infrastructure node. The build system is contained inside a container; therefore, the infrastructure part of the testbed device stays unaffected by any build failures or run-away processes. If the building process succeeds, the infrastructure node in Step 7 flashes the target node with freshly built firmware and in Step 8 executes the test process. In Step 9, the infrastructure node obtains the results from the target node and examines them. In Step 10, the automation system updates the repository with the result and communicates the result to the developer, typically via email.
3 COINS architecture and implementation
The architecture of COINS is illustrated in Figure 2b and further explained in this section. It consists of the three main architectural components identified and described in Section 2, which are mapped on the elements of the existing LOG-a-TEC infrastructure [7, 8].
For the reference implementation of COINS
3.1 Testbed devices
The LOG-a-TEC testbed devices can use several wireless technologies, as depicted on the left side of Figure 2b. Architecturally, we consider hybrid testbed devices with two separate functional blocks represented by the infrastructure node and the target node. The main reason for the separation of the testbed device into two functional blocks is to have a generic infrastructure node that can be combined with various target nodes to support a range of heterogeneous experiments with a unified way of management and reconfiguration. Another reason is to achieve hardware separation between robust and stable testbed management functionality on the infrastructure node and the experimental code, which is prone to errors and crashes on the target node. The infrastructure node can usually also support containerization, while the target node must often be based on an embedded system with little hardware support for process isolation features due to the requirements of the software under test.
In our implementation, the infrastructure node is a custom-designed single board computer based on the BeagleCore module running the Debian GNU/Linux operating system. It features wired Ethernet and Wi-Fi for infrastructure connectivity, an SD card slot for storage expansion, VESNA and USB interfaces for modular extensibility and a development/debug process. The target node is a custom VESNA (VErsatile platform for Sensor Network Applications) device with an ARM Cortex-M3 microcontroller application module and dedicated experimentation transceivers. VESNA can run a dedicated OS (e.g., Contiki) or custom firmware. The integration of the two is done via custom hardware design and the adoption of selected interfaces and protocols, thus going beyond similar approaches such as . Both nodes are interconnected via application and development interfaces, providing the exchange of application data as well as remote low-level application debugging. The development interface is based on JTAG and the application interface on protocols such as LCSP (Light-weight Client Server Protocol; an HTTP-inspired protocol ) running on top of a serial interface.
The repository support was added to the system to abstract the underlying complexity and enable the CI process described in Section 2. It includes (i) the target test that has to be executed on the testbed device; (ii) the test control that runs on the management server and communicates with the testbed device; (iii) the developer’s code that has to be tested; (iv) dependencies such as software libraries; (v) the container file, which describes how to package the target test, test control, dependencies and developer’s code into a container; and (vi) the deployment configuration, which specifies where and how to build the container.
A minimum test case for a wireless communication system includes software for three entities: the transmitter, the receiver and the test controller. The first two will run on the target node within the testbed device, while the third runs on the management server. The test controller instructs one testbed device to go in receive mode and another testbed device to transmit the test sequence. When the first testbed device receives the test sequence, it reports the results back to the test controller, which asserts that the received sequence matches the transmitted one. If it does, the test succeeds; otherwise, it fails.
In our implementation, the target test and test control are realized using C and Python, but in general, the proposed framework is programming language independent. The container support is implemented through Docker, while the deployment is automated by Ansible. The repository that contains everything needed for the completely automated build is realized through a GitHub repository. Examples of more advanced tests and experiments are summarized in Section 4.
3.3 Infrastructure management and build automation system
The infrastructure management and build automation system is a complex setup composed of self-sufficient systems packaged in a Docker container following the micro-service architecture approach. It contains (i) the node registry, which has a list with details about all testbed devices; (ii) the node monitoring service, which is in charge of monitoring the activity and health of the testbed devices; (iii) the orchestration and configuration, which performs regular maintenance and reconfiguration of the testbed; (iv) the CI hook service, which waits for signals of changes in the repository and initiates the build process; and (v) the container file, which describes how to package all the previous components in a container.
There are several periodic background jobs running on the management system that perform house-keeping tasks:
building the Ansible host file from the node registry database,
updating the information on the availability of individual nodes using the Ansible ping command,
stand-alone testbed device monitoring based on Munin, and
CI Hook service triggering the build process.
Externally, the management server exposes a user-facing web interface and an HTTP REST API . The infrastructure management and build automation system allows browsing through clusters of individual devices and nodes in the database, positioning devices on a map, visualizing reports, monitoring device activity, etc. Thus, it gives an instant overview of the state of the testbed at any time. Warnings can be defined to be triggered when specific monitored variables exceed a given threshold, such as running out of system memory or storage space. In such cases, the system sends an email to the system administrator. Depicting the testbed devices on a map is particularly useful for the selection of devices for different test cases. Programmatic access to the management system is performed through the REST API; i.e., nodes use the REST API to perform automatic registration and update changes in the configuration.
4 Testbed deployment and supported functionality
In order to enable testing and experimentation with the COINS framework in a dense and heterogeneous 5G capillary radio environment, we upgraded the combined outdoor/indoor LOG-a-TEC testbed at the JSI campus with the necessary tools and services described in Section 3. Table 1 summarizes the number and type of testbed devices in an indoor and outdoor environment, whereas the layout of devices in the outdoor environment is depicted in Figure 3.
|Node type||Radio||Frequency||No. of dev.|
|Target node SRD A||Atmel AT86RF212||868 MHz||21 outdoor|
|TI CC2500||2.4 GHz|
|Target node SRD B||TI CC1101||868 MHz||21 outdoor|
|Atmel AT86RF231||2.4 GHz|
|Target node LPWA||Semtech SX1272||868 MHz||3 outdoor|
|Target node UWB||DecaWave DW1000||3.5 - 6.5 GHz||11 outdoor|
|(6 channels)||20 indoor|
|Target node||NXP TDA18219HN||470 - 866 MHz||2 outdoor|
|UHF spec. sens.|
|Infrastructure||TI WL1837||5 GHz||58 outdoor|
The outdoor part of the wireless testbed is located in and around a park area of 55 m by 60 m. Testbed devices are placed on light poles 3.5 m above the ground and on the surrounding buildings at heights from 2.0 to 9.3 m. To enable experiments in indoor as well as in mixed indoor/outdoor scenarios, the testbed is extended in an indoor environment with additional 20 UWB devices and one LPWA device. These are deployed on the second and third floors of the 28.4 m by 16.6 m building. In addition, several portable devices of all types described in Section 3.1 are at disposal for analyzing the use cases incorporating device mobility or further network densification.
The diversity of types, the relatively high number and density of testbed devices, and the possibility of hosting multiple radio technologies in most devices to transmit/receive/sense at the same time at the same location enables experimentation scenarios in a very dense and heterogeneous wireless environment. The various types of experiments include: (i) investigation of assisted and non-assisted indoor/outdoor localization algorithms; (ii) spectrum sensing for dynamic spectrum access and cognitive radio networking; (iii) link and/or network performance evaluation in a controlled interference and/or heterogeneous radio environment; (iv) design and evaluation of new custom-developed wireless protocols and their benchmarking against the standardized protocols; and (v) performance evaluation and controlled piloting of prototype IoT devices.
In order to support such diverse testing capabilities, the repository comprises a set of testbed libraries and tools. They enable: out of the box localization; Line-of-Sight/non-Line-of-Sight link classification; link quality estimation (LQE), classification and prediction; custom protocol stack development using modular components; spectrum sensing by energy detection, cyclostationary detection and eigenvalue- and covariance-based methods; signal generation including wireless microphone profiles; and game theoretic power allocation. These software components and modules can be incorporated in custom developed algorithms or adapted to the particular use case and subsequently used as a testing benchmark. They can also be used for setting the required testing environment. In particular, for evaluation and comparison of spectrum sensing algorithms and other wireless technologies or algorithms, the testbed enables generating controlled interference. This also helps with the reproducibility of tests and enables evaluation of interference specific performance of various higher-layer protocols. Furthermore, localization and distributed spectrum sensing can be used to build geolocation spectrum occupancy databases, which in turn can be used in investigation of primary/secondary spectrum access regimes.
5 Related work
While we did not find any other framework addressing wireless experimentation with CI support, we have identified three existing frameworks that are to some extent similar to COINS, i.e., the cOntrol and Management Framework (OMF) , the Network Implementation Testbed Laboratory (NITOS)  and the Berlin Open Wireless Lab (BOWL) . We performed a feature-wise comparison of these frameworks and summarized it in Table 2.
In the comparison, we considered an extensive list of features that can be used to compare the experimentation systems . The list of features, however, lacks the CI-specific properties that our work focuses on; therefore, we extended the list with four core CI concepts [15, 16]:
Existence of a single source repository that contains everything needed for the completely automated build process.
Support for completely automated build process executed on each commit.
Self-tests included inside the repository, which run on each build.
Fast build process so that each commit can be built and tested.
For each feature presented in Table 2, we explain how we addressed the problem compared to the other frameworks. The last four features are CI specific and are realized only in COINS.
The description language is the language used to describe how and where to build and run the experiment. In OMF, the experiment is described using OMF Experiment Description Language. The NITOS testbed is abstracted through a REST API. The BOWL system is based on the OpenWRT Unified Configuration Interface (UCI) scripts. The COINS framework is based on well-known tools that many developers use in their daily development practices. We did not want to introduce a new language specifically for this purpose. Instead, we rely on Ansible and Docker to configure what to run where, how to obtain results, etc.
The type of experiment describes the platform where the experiment is going to run. OMF and NITOS can support wired or wireless experiments, whereas BOWL is limited to WiFi-based wireless systems. COINS focuses on wireless systems and is suitable also for restricted capability devices. While OMF, NITOS and COINS are suitable for developing completely new as well as optimizing existing wireless solutions, BOWL is primarily suitable for optimizing existing solutions.
The operation refers to the ability of the testbed to also serve as a production system. OMF and NITOS are exclusively testbed frameworks, while BOWL is designed to work simultaneously as a testbed and production WiFi access network. In this respect, COINS is also exclusively a testbed framework.
Framework interoperability covers an infrastructure independent design, support for existing infrastructure services, resource discovery and software interoperability. OMF and NITOS address interoperability issues, while BOWL UCI configuration scripts are implementation specific. OMF can be extended, and NITOS specifies a testbed API. COINS addresses interoperability by offering an implementation independent design.
Reproducibility refers to the ability to automatically run the same experiments multiple times. OMF enables complete reproducibility, while NITOS offers limited reproducibility since it implements automated node selection based on availability. BOWL also addresses the problem of making experiments reproducible. COINS supports complete reproducibility by creating tags inside the repository and manually choosing the devices used in the test. While declaratively all frameworks address reproducibility, we argue that it may be severely affected in case of missing repository support on the framework level.
Fault tolerance addresses recovery capabilities when the experiment does not execute as expected. OMF and NITOS frameworks are designed to be fault tolerant. In OMF, there is a Management Plane responsible for rebooting a resource and setting up particular disk images on PC-based resources. NITOS uses an automatic monitoring tool to diagnose a malfunction in one of the resources. BOWL is not fully fault tolerant as it also serves as a production WiFi network for actual users. COINS is inherently fault tolerant due to the use of container technology to isolate the experiment environment from the infrastructure. Furthermore, it compiles the firmware from source code each time and uses JTAG for flashing.
The debugging process addresses the ability to identify problems and their causes during experiment execution. All compared frameworks are designed to produce extensive log files during the build process. However, COINS does not rely on a single log file but rather a log repository, which does not have a fixed structure. Thus, the developer designing the build process can place different files inside this repository, which is published together with the build results.
Monitoring is a feature that enables the developer or a maintainer to oversee the status of the system at any time. All compared frameworks have a system for monitoring. COINS uses the Ansible ping method, which works on top of SSH to identify which nodes are accessible. Furthermore, it integrates a standalone monitoring tool Munin as part of the management system.
Data management is responsible for the distribution and collection of data. OMF uses the Measurement Collection system based on a relational database. NITOS and BOWL do not prescribe any dedicated data collection mechanism on the framework level. COINS uses the same system for build data management as for debugging; i.e., each build produces the result files along with the debug files in the build repository.
The COINS framework is designed to keep everything needed for a completely automated build inside a single version-controlled source repository. This is in line with one of the fundamental CI concepts.
The automated build is another core CI concept. This system ensures that a new code committed by the developer does not break the process, which is particularly important in CI. We showed that COINS defines the automated build system, which deploys the modified source code to the appropriate targets where it is built and tested.
Tests of each committed change are extremely important for the CI process. In COINS, all the tests are included inside the repository. The test process is executed automatically on each commit, and the result is communicated to the developer.
The last core CI concept requires a fast build process so that the developer does not have to wait long for the test results. To achieve fast builds, COINS relies on the container technology and incremental changes delivery to the testbed device. Therefore, only the changes are rebuilt within the container, thus making the build process fast.
6 Open Challenges
For wider adoption of CI practices in the development and testing of wireless technologies, we identified three groups of challenges. The first group revolves around the constraints posed by the physical environment; the second one is concerned with process automation; and the third group is related to the lifecycle management of the wireless product.
6.1 Challenges related to the physical environment
A CI system for wireless networks comprised of distributed hardware has to take into account that there may be unpredictable radio interference on the physical transmission medium or failures in hardware, and it must be ensured that the test reports to the developer are immune to them. In other words, from the test reports, it should be clear if the test failed because of software issues or because of conditions in the physical environment.
A possible approach for ensuring reproducibility is for the CI framework to use spectrum sensing to identify a suitable test channel and thus avoid interference. Additionally, using controlled interference in test cases can sometimes prove useful to verify the performance of a particular wireless algorithm.
A possible approach for ensuring reliability is to have redundancy in terms of distributed hardware and perform the same tests on more disjoint subsets of testbed devices.
6.2 Challenges related to the process automation
With the implementation of the proposed framework, it is possible to realize automated system tests. However, while the CI system and process are fully automated, the detection of errors caused by the physical environment is currently manual. This is an issue specific to wireless environments and has yet to be rigorously addressed. Additionally, the developer of the test currently needs to manually select one or more testbed devices to be used in the test.
A possible approach for enabling fully automated tests is to design and develop a mechanism that would analyze spectrum sensing data in correlation with the tests and repeat the test if it failed because of a temporally jammed channel. This process could be made transparent to the developer.
Automated resource allocation
A possible approach for enabling automated resource allocation is to develop a mechanism that would automatize the resource (i.e., testbed device) assignment for individual test execution. It would be interesting to see if the concurrent tests could be somehow categorized and how this information could be utilized by an algorithm for automatic selection of testbed devices to optimize test execution and testbed utilization.
6.3 Challenges related to the lifecycle management of the wireless product
Continuous software development should also consider continuous delivery of the builds in production. The proposed framework is focused on developing quality software also for restricted capability devices. However, additional consideration could be given to extending the framework with continuous delivery capabilities to solve the issue of wireless devices, which are not maintained after entering the market. Continuous delivery could also solve security issues in devices on the market. Currently, every electronics producer addresses the problem of continuous delivery separately; however, it would be beneficial to find some standard method.
In this paper, we introduced continuous integration (CI) practices for wireless network development. We proposed the COINS framework, which simplifies the code development process on real wireless testbeds and enables frequent code changes. We then showed that the proposed framework can be implemented on a testbed composed of state-of-the art hardware modules suitable for 5G capillary wireless MTC experimentation. We realized a reference implementation of COINS by integrating existing open-source components and cloud services into a fully functional system supporting CI for wireless networks. We performed a qualitative comparison of the proposed framework with three related existing frameworks. The comparison uses a list of features already used in the community for comparing experiment management tools. However, to fully illustrate the advantage of the proposed framework, we had to extend the available list of features with four CI-related ones. Finally, we identified and discussed open challenges.
This work was partly funded by the Slovenian Research Agency (Grant no. P2-0016) and the European Community under the H2020 eWINE (Grant no. 688 116), WiSHFUL (Grant no. 645 274) and Fed4FIRE+ (Grant no. 732 638) projects.
MATEVŽ VUČNIK received his B.Sc. in electrical engineering from the University Of Ljubljana, Faculty of Electrical Engineering in 2010. He is currently a PhD candidate at the Jožef Stefan International Postgraduate School and works as a senior research assistant in the Department for Communication Systems at the Jožef Stefan Institute. His main interests include distributed software development and Web technologies, focusing on communication protocols and IoT integration. He has actively collaborated on several EU projects.
TOMAŽ ŠOLC studied electronics at the Faculty of Electrical Engineering, University of Ljubljana and received his B.Sc. in 2007. He is currently a senior research assistant at the Department of Communication Systems, Jožef Stefan Institute and a Ph.D. student at the Jožef Stefan International Postgraduate School. His work involves hardware design, measurements and embedded software development as well as research in the field of spectrum sensing. He participated in various national and EU projects.
URBAN GREGORC received a Master degree in Electrical Engineering from the University of Ljubljana, Faculty of Electrical Engineering in 2016. After graduation, he worked at the Department of Communication Systems at the Jožef Stefan Institute, Ljubljana as a research assistant and in 2018 moved to a startup ComSensus. His research is focused on Low Power Wide Area Network technologies for the Internet of Things.
ANDREJ HROVAT received his B.Sc. in 2004. He obtained a Ph.D. degree in Electrical Engineering from the Jožef Stefan International Postgraduate School, Slovenia in 2011. He is currently a researcher in the Department of Communication Systems of the Jožef Stefan Institute and an assistant professor at the Jožef Stefan International Postgraduate School. His research interests include radio signal propagation, channel modelling, terrestrial and satellite fixed and mobile wireless communications, radio signal measurements and emergency communications.
KLEMEN BREGAR received a degree in electrical engineering from the University Of Ljubljana, Faculty of Electrical Engineering in 2013. Since 2013, he has worked as a young researcher and Ph.D. candidate at the Department of Communication Systems at Jožef Stefan Institute, Ljubljana. His research interests include indoor localization, ultra-wide band communication systems, biomedical wireless sensor networks and machine learning.
MIHA SMOLNIKAR received his BSc from the University of Ljubljana and has more than 10 years of experience in networked embedded systems and wireless communications. His main interests include wireless sensor networks, energy harvesting, sensor technologies and the optimization of the protocol stack. In the area of IoT, he focuses on the development of products and services for the smart home, smart city and smart grid segments. He actively participates in several EU projects.
MIHAEL MOHORČIČ is head of the Department of Communication Systems and Scientific Counselor at the Jožef Stefan Institute. He is also associate professor at the Jožef Stefan International Postgraduate School. His research experience include development and performance evaluation of network protocols for mobile and wireless communication systems, cognitive radio networks, wireless sensor networks, dynamic composition of communication services and wireless experimental testbeds. He has participated in many EU projects. He is a Senior Member of IEEE.
CAROLINA FORTUNA received her B.Sc. in 2006, Ph.D. in 2013 and was a postdoctoral research associate at IBCN, Ghent University, 2014-2015. Currently, she is research fellow at the Department of Communication Systems, Jožef Stefan Institute and an assistant at the Jožef Stefan International Postgraduate School. Her research is interdisciplinary, focusing on data and knowledge driven modelling of communication and sensor systems. She has participated in EU and industrial projects with Bloomberg LP and Siemens PSE.
- M. Condoluci, M. Dohler, G. Araniti, A. Molinaro, and K. Zheng, “Toward 5G densenets: architectural advances for effective machine-type communications over femtocells,” IEEE Communications Magazine, vol. 53, no. 1, pp. 134–141, 2015.
- E. De Poorter, J. Hoebeke, M. Strobbe, I. Moerman, S. Latré, M. Weyn, B. Lannoo, and J. Famaey, “Sub-ghz lpwan network coexistence, management and virtualization: An overview and open research challenges,” Wireless Personal Communications, vol. 95, no. 1, pp. 187–213, Jul 2017. [Online]. Available: https://doi.org/10.1007/s11277-017-4419-5
- S. Wahle, C. Tranoris, S. Denazis, A. Gavras, K. Koutsopoulos, T. Magedanz, and S. Tompros, “Emerging testing trends and the Panlab enabling infrastructure,” IEEE Communications Magazine, vol. 49, no. 3, pp. 167–175, March 2011.
- M. Meyer, “Continuous integration and its tools,” IEEE software, vol. 31, no. 3, pp. 14–16, 2014.
- B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov, “Quality and productivity outcomes relating to continuous integration in GitHub,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 2015, pp. 805–816.
- J. Pinola, J. Perälä, P. Jurmu, M. Katz, S. Salonen, J. Piisilä, J. Sankala, and P. Tuuttila, “A systematic and flexible approach for testing future mobile networks by exploiting a wrap-around testing methodology,” IEEE Communications Magazine, vol. 51, no. 3, pp. 160–167, 2013.
- T. Šolc, C. Fortuna, and M. Mohorčič, “Low-cost testbed development and its applications in cognitive radio prototyping,” in Cognitive radio and networking for heterogeneous wireless networks. Springer, 2015, pp. 361–405.
- C. Fortuna, A. Bekan, T. Javornik, G. Cerar, and M. Mohorčič, “Software interfaces for control, optimization and update of 5G machine type communication networks,” Computer Networks, 2017.
- T. Buchert, C. Ruiz, L. Nussbaum, and O. Richard, “A survey of general-purpose experiment management tools for distributed systems,” Future Gener. Comput. Syst., vol. 45, no. C, pp. 1–12, Apr. 2015. [Online]. Available: http://dx.doi.org/10.1016/j.future.2014.10.007
- J. S. C. Xiao, S. K. V. Ragavan, and M. Shanmugavel, “OSGi-based, embedded, distributed, telematics framework for flexible service provisioning in cyber-physical production systems,” in 2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Dec 2016, pp. 1–5.
- T. Rakotoarivelo, M. Ott, G. Jourjon, and I. Seskar, “Omf: A control and management framework for networking testbeds,” SIGOPS Oper. Syst. Rev., vol. 43, no. 4, pp. 54–59, Jan. 2010. [Online]. Available: http://doi.acm.org/10.1145/1713254.1713267
- D. Stavropoulos, A. Dadoukis, T. Rakotoarivelo, M. Ott, T. Korakis, and L. Tassiulas, “Design, architecture and implementation of a resource discovery, reservation and provisioning framework for testbeds,” in 2015 13th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), May 2015, pp. 48–53.
- T. Fischer, T. Hühn, R. Kuck, R. Merz, J. Schulz-Zander, and C. Sengul, “Experiences with bowl: Managing an outdoor wifi network (or how to keep both internet users and researchers happy?),” in Proceedings of the 25th International Conference on Large Installation System Administration, ser. LISA’11. Berkeley, CA, USA: USENIX Association, 2011, pp. 24–24. [Online]. Available: http://dl.acm.org/citation.cfm?id=2208488.2208512
- M. Vögler, J. Schleicher, C. Inzinger, S. Nastic, S. Sehic, and S. Dustdar, “LEONORE – Large-Scale Provisioning of Resource-Constrained IoT Deployments,” in 2015 IEEE Symposium on Service-Oriented System Engineering, March 2015, pp. 78–87.
- M. Fowler, “Continuous integration,” http://www.martinfowler.com/articles/continuousIntegration.html, 2006, accessed on 10.11.2017.
- D. StÃ¥hl and J. Bosch, “Modeling continuous integration practice differences in industry software development,” Journal of Systems and Software, vol. 87, no. Supplement C, pp. 48 – 59, 2014. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0164121213002276