Observations on OMNeT++ Real-Time Behaviour
OMNeT++ is a widely used platform for all types of network simulations. The open source simulation framework Artery can be used to perform Vehicular Ad Hoc Network (VANET)simulations. This paper presents an approach for connecting this simulation and real-world VANEThardware to extend the test range and investigates the real-time behaviour of the simulation. As a Device Under Test (DUT)depends on real-time data to perform properly, different simulation scenarios running different hardware setups are presented. Additionally, the paper deals with the impacts of real-time losses on the test run outcomes. Most time dependant algorithms like the duplicate packet detection do not need very accurate real-time data and thus could be verified using the presented approach. Otherwise, in some cases such as testing of multi-hop communication, accurate real time is crucial.
In times of increasing complexity of advanced driver assistance systems, it is crucial to enhance the environmental awareness of vehicles. Vehicles can be equipped with Vehicular Ad Hoc Network (VANET) devices, acting as a new information source besides already well known sensors.
VANETSare spontaneously created networks between road participants and Road Side Units (RSUs). They are based on IEEE 802.11p  and ETSI ITS G5  in Europe as well as IEEE WAVE in the USA . This paper focuses on the European ETSI ITS G5 standards.
VANETSand their applications were developed with the focus on enhancing traffic safety and traffic flow . Especially in critical driving situations, availability of information is crucial. This leads to the question, how to test VANETcommunication properly. As Software in the Loop (SIL)simulation is not enough to ensure the availability, this paper evaluates if Artery can fulfil the real-time requirements necessary for Hardware in the Loop (HIL)testing.
In section II a overview of the related testing frameworks and hardware testbeds is given. Section III presents the state of the art and the theoretic concept of the hardware testbed. In section IV different scenarios running on different hardware setups are investigated. Section V presents the limitations of the presented approach. Finally, section VI includes a conclusion and a brief outlook.
Ii Related Work
Currently there are mainly two different Inter-Vehicular-Communication (IVC)testing approaches. On the one hand, there is pure software testing with frameworks like Artery111https://github.com/riebl/artery  and Veins222http://veins.car2x.org  which are based on the discrete event simulator OMNeT++. These implement the Intelligent Transport System (ITS)G5 and the Wireless Access in Vehicular Environments (WAVE)standards, respectively. Also, different environmental circumstances and accidents can be modelled easily .
On the other hand, there are real-world testing approaches like the "Testfeld A9" established near Ingolstadt in Germany. Real-world testing allows for testing actual IVChardware, but it is required to have at least two drivers in real vehicles equipped with IVChardware. Thus, field tests are hard to reproduce and very expensive. While it is still possible to perform simple real-world scenarios, like presented in , it might be not feasible to do this with more complicated scenarios.
HILtests can be used to perform hardware tests which are easy to repeat and cost effective. It is not a new approach using OMNeT++ to connect a simulation with real-world hardware as OMNeT++ was one of the fastest simulators in the domain of wireless networks in prior software versions . In , a routing framework for OMNeT++ HILsimulations is presented and the real-time behaviour of OMNeT++ was investigated in different scenarios. They mentioned that real time will be a problem in future scenarios, especially if the node topology begins to change dynamically due to node mobility. According to , OMNeT++ can also be coupled with RoSeNet to connect with real-hardware sensors. In  the performance of the INET framework (release 2.5.1) in emulation mode combined with an enhanced real-time scheduler was investigated. They pointed out to have performance problems. Additionally, they observed packet losses occurring in the communication between the real hardware and the simulation.
Iii HiL concept
Artery is an open source framework for the discrete event simulation OMNeT++ . It allows for simulating European VANETsusing Vanetza333http://vanetza.org/, which is an open source implementation of the ETSI ITSG5 communication stack . Veins or INET provide the physical and Medium Access Control (MAC)layers. Moreover, the movement of the vehicles is simulated by the open source traffic simulator SUMO444http://sumo.dlr.de/. SUMO and Artery are coupled using the Traffic Command Interface (TraCI).
Iii-a Artery Overview
Basically, each vehicle controlled by SUMO is represented by an OMNeT++ compound-module called Car. A Car module is, among others, composed of a Middleware, a VanetNic and a Mobility submodule. The Mobility module is responsible for all vehicle dynamics related data and information. The VanetNic represents the network interface of each car. The Middleware module hosts all registered applications and contains an instance of Vanetza, taking care of routing and transport the incoming and outgoing IVCpackages. Currently, the day one applications Cooperative Awareness Message (CAM)and Decentralized Environmental Notification Message (DENM)are provided by Artery [5, 15].
COOPERATIVE AWARENESS (CA)is responsible to inform all road users in the network about basic data of all other participating vehicles. This basic data contains, among others, the network node’s position, speed and heading. CAMSare generated up to ten times per second from each road participant . In contrast, DENMsare only generated in case of a specific event. A DENMcontains information of hazards and dangerous situtations occurring in road transport . Thus, Artery allows for defining this dangerous situations and environmental influences like, for example, heavy rain or nearly accidents . These situations are required to trigger related DENMmessages.
Iii-B HiL concept
Figure 1 shows the basic concept of a HILsimulation using Artery to provide test data. The red coloured vehicle is the simulated representative of a Device Under Test (DUT). For now, this car will further be called physical twin. If a message is going to be transmitted to the physical twin, the message will be transferred to a 802.11p gateway realised on software defined radio technology. This behaviour is represented by red arrows, showing the connection of the simulation to the real word and thus with the DUT. The blue dotted arrows show the current behaviour of the simulation.
In contrast to other simulated cars, the physical twin has a quite limited functionality: Upper protocol layer processing is done by the DUTso the Middleware module as well as the Vanetza instance can be dropped. Otherwise, the Mobility module is still needed to provide Global Positioning System (GPS)data to the DUTto ensure properly working routing protocols . Without appropriate GPSinformation, the geographic routing protocols defined in  would not work properly. Summarising, all functionality beginning with the MAClayer processing is stripped from the physical twin. It is only responsible for calculating message receptions and feeding received messages back on the OMNeT++ channel.
As each kind of hardware testbed is dependent on real-time execution, the simulated environment must run in real time, too. To ensure this, the OMNeT++ built-in scheduler is exchanged by a real-time scheduler. This scheduler is built upon Boost ASIO timers, ensuring asynchronous waiting. Hence, the real-time scheduler slows down the simulation, if it could run faster than real time. Additionally, it is aware of real-time losses so that it could stop the simulation if data could not be provided in real time. Also, the scheduler provides logging mechanisms to investigate the real-time behaviour of simulation runs after they are finished. A pseudocode implementation of the scheduler can be found in Appendix A.
Iv Investigations on Real-Time Behaviour
|Component||Laptop Computer||Simulation Cluster|
|CPU||Intel Core i5-6300U @ 2.40GHz||Intel Xeon E7-8867 v4 @ 2.40GHz|
|Cores||1 x 4||4 x 18|
|Hard Drive||256GB SSD||450GB SAS SSD RAID 1|
|ID||Event name||# Events|
|"3 vehicles"||# Events|
|4||GeoNet data frame||3870||11298|
|9||remove non Interfering Transmission||661||1188|
|13||GeoNet radio frame||1274||4460|
As already known from investigations described in  there is a quite low execution speed of OMNeT++ VANETsimulations. This leads to the question, if Artery is capable of reaching and holding real time to provide data for a HILtestbed while doing an online simulation.
Iv-a Scenario Description
The chosen scenario is very simple: Three vehicles driving on a highway from north to south.
As the amount of driving vehicles seems to be the main influence on the execution speed of these simulations, the second test scenario increases the amount of driving vehicles by two. All other parameters remain the same as in the previous scenario. The setup of the radio medium is configured by Artery using its INET defaults. The in Section III-B introduced scheduler is used to evaluate the real-time behaviour. The simulation was built using OMNeT++ version 5.1.1 and was executed on two different computers: A laptop computer and a simulation cluster. Table I presents a comrehensive hardware list of the used computers. As OMNeT++ uses only one core in this test setup, the speed-up of the simulation running on the simulation cluster is caused by faster calculation hardware not by better parallelism.
Iv-B Simulation Results and Scenario Comparison
Table II presents the number of events occurring in both scenarios. The ID column is used to map event IDs to the events depicted in the box plots in figure (a)a and (a)a. The third and the fourth columns show the number of occurring events dependent on the simulated scenario. Thus, compared to the base scenario which triggers 16430 events, the scenario with two more vehicles triggers 42315 events. Hence, adding two cars causes 2.58 times more events in this scenario setup.
Figure 3 and 3 present the evaluations of both scenarios executed on the simulation cluster and the laptop computer. The included boxplots depict the execution time of events. The indices on the x-axis correspond to the events mentioned in Table II. If we compare both figures, the time a event needs to be executed is lower while using the simulation cluster but relative event durations remain nearly the same. This behaviour is caused by the faster computing capabilities of the single cores of the simulation cluster.
The histograms (b)b and (c)c show the real-time misses per event while the scenarios were executed on the laptop. Histogram (b)b and (c)c present the same but for the execution on the simulation cluster. As the green bars indicate events executed nearly in real time, higher green bars and lower red bars indicate a nearly real-time capable simulation run.
It can be seen that there are real-time drops up to 1.5 seconds in each scenario. This is caused by the TraCi Connect event, which is executed as first event and takes about one second. Thus, even if the scenario itself is real-time capable, there will be always a few seconds at the beginning of the simulation which must be skipped because of the system startup. Figure (b)b is a great example for a good and fast running scenario. There are only roughly 50 - 100 events which are more than one second behind the real time. About 15000 events out of the total amount of 16430 events are executed nearly in real time. This means, the simulation cluster can basically handle a small scenario with three vehicles in real time.
Figure (b)b presents the same scenario executed on a laptop computer. It can be seen that there are still about 10000 events executed nearly in real time. Other 5000 events are executed only 0.1 second behind real time. Thus, only a few more events can be found in the area of about one second behind real time, but the scenario looks still quite good.
Distinct differences between execution speed of the computing hardware can be seen in Figure (c)c and Figure (c)c. The simulation cluster is able to handle the five car scenario with still up to 1.5 seconds real-time loss. The amount of events in the area from 1 to 1.5 seconds is significantly increased, compared to the base scenario but OMNeT++ is still able to catch up. However, the five car scenario executed on the laptop computer exceeds the 1.5 seconds limit by far. There are many real-time losses up to three seconds. Thus, the five car scenario can only be handled by the simulation cluster.
Figure 4 admits a closer look on the real-time behaviour of the different simulation runs. Blue bars show the time needed to process all events occurring at a particular simulation time stamp. The red line depict the current gap between simulation time and real time. Thus, a higher blue bar causes a higher real-time loss peak. It seems that Artery tends to schedule a bunch of events every 50ms. This behaviour causes the simulation to fall behind real time every 50ms, which is indicated by the peaks of the red line.
As it can be seen, the simulation cluster produces as many real-time losses as the laptop computer. But, even if the differences between the execution time of single events are not that clear, the real-time losses are much higher on the laptop computer, independent from the executed scenario. This approves that Artery has to execute many events at the same time because they are scheduled at nearly the same timestamp. Due to the lack of parallelism, OMNeT++ has to execute this events in a sequence, which causes the real-time losses.
As there are many events which are closely linked together, for example the GeoNet packet and the GeoNet data frame, this behaviour could not be changed significantly. Thus, there will always be real-time losses caused by many events occurring at the same time if there is no parallelism.
Iv-C OMNeT++ Time Flow
Figure 5 depicts a generic example how the timeline in OMNeT++ behaves compared to the real-time flow.
As it can be seen, the absolute time needed to execute one simulation millisecond is sometimes higher than one real millisecond like indicated at millisecond two. If the simulation is behind the wall-clock time, it has to catch up to avoid higher real-time gaps. Hence, the simulation is trying to execute events faster than in real-time till simulation time and wall-clock time are matching again. But, even if the OMNeT++ clock sometimes progresses faster than real time, the fixed time stamp is always behind or equal to the wall-clock time which is ensured by the real-time scheduler. If the next event is located in the future, the scheduler waits till the timestamp of the event matches the wall-clock time. Thus, it is ensured that the DUTmust only deal with timestamps in the past and not in the future, which would be much more problematic.
V Simulation Result Interpretation
Most scenarios in IVCcommunication do not depend on very accurate timestamps. For example, the Long Position Vector (LPV)[18, Section 8.8] contains the time at which the last geographic position update was received by the vehicle writing the LPV. As in a real-world environment GPSupdate rates are fluctuating, the timestamp in the LPVis not strongly related to the current wall-clock time.
The LPVtimestamp is used to perform duplicate packet detection at the network layer [18, p. 63]. The duplicate packet detection algorithms depend on the sequence number as well as the timestamp. A packet is identified to be a duplicate if the timestamp is lower or equal to an already received packet. As the introduced RealTimeScheduler ensures that the simulation time is always behind or equal to the wall-clock time, this algorithm works like expected.
CAMScontain a generation timestamp  used, for example, to recognise and avoid replay attacks [2, Section 7.6.1]. However, a CAMis not detected to be a duplicate if there is only a slight difference between wall clock time and the generation timestamp.
A DENMcontains various timestamps as well. There is, for example, an expiry time after which a DENMevent is terminated. Mostly, this period lasts several seconds but in cases like the dangerous situation  trigger, a DENMevent lasts only two seconds. Thus, in some situations a real-time loss higher than one ore two seconds may invalidate a DENMwrongly.
Another issue is the long-range communication using multiple vehicles as forwarders . A MAClayer unicast is followed by an Acknowledgement Frame (ACK), sent by the receiver, to confirm a successful packet transmission. The time span in which the ACKis expected by the initial sender is the time of a Short Interframe Space (SIFS)[1, Section 188.8.131.52]. Determined by the channel bandwith of 10 MHz  the SIFSis 32 [1, Table 18-17]. This very short time span leads to a potential problem: If a MAC layer unicast is sent to the DUT, it has to respond with an ACKwithin this SIFS. As depicted in Figure 5 between millisecond two and three, OMNeT++ could run faster than real time. Thus, the sending vehicle inside OMNeT++ may not receive this message in the claimed timespan. Hence, this problem must be solved to perform multi-hop tests.
Summarizing, in most cases, a real-time loss in a range of a few milliseconds is not that problematic. Important algorithms like the duplicate packet detection do only rely on linear time flow and do not depend on hard real time. Only a few scenarios like multi-hop testing relay on a very accurate time synchronisation. Thus, Artery is basically able to provide online simulation data for a real-time testbed.
This paper presented an approach for extending Artery to provide hardware tests. Artery is used to facilitate SILtests in the area of VANETs. As a HILsimulation always depends on real-time data, the online simulated scenarios have to be executed in this manner. It was investigated that this criteria can only be fulfilled if only a few cars are simulated. Also, even if the simulation is overall real-time capable, there are always real-time losses. How significant these losses are depends on the used computation hardware and the scenario complexity. As OMNeT++ only uses one core when simulating VANETs, a higher single core performance causes a higher execution speed. Also, Artery and other wireless communication models tends to produce events to be executed nearly at the same time, causing more significant real-time losses. This is related the fact that one sending event triggers various receiving events.
Also, it was audited in which situations nearly hard real time is required and when real-time drops are bearable: Multi-hop communication strongly depends on real time data, so its not possible to test this feature properly in the current state of the simulation. Most other algorithms do only depend on a steady time flow and are not influenced by real-time losses in the range of a few milliseconds. This is the case for DENMmessage expiries or the recognition of replay attacks. Also the duplicate packet detection is not affected harmfully.
In conclusion, Artery could basically be used to provide data for VANETHILtests. In case of multi-hop-test scenarios, real-time losses may influence the test results heavily. Other scenarios can be used to provide functional testing of VANEThardware. This scenarios are currently limited to three or four vehicles simulated vehicles depending on the used hardware.
As this paper presents a work in progress research project, the HILtestbed will be created with the observed behaviour of OMNeT++ in mind. Thus, to enable multi-hop testing, which is crucial in European VANETs, one idea is to use a Software Defined Radio (SDR) as 802.11p proxy. This allows for a modified MAClayer of the used proxy device to send ACK frames depending on the current state of the simulation. Hence, the proxy device must know all vehicles which can communicate with the DUTin the current simulation step. This idea could provide a basic implementation of a testbed which can handle most functionality of ETSI ITSG5 networks with the constraint that the simulation must run in nearly real time.
So, further work must be done in the field of enhancing simulation speed to achieve faster running simulation scenarios. Moreover, other ways to provide test data for hardware tests can be investigated. This includes, among others, the capturing of the simulated network traffic and playing them back in real time.
Appendix A Real Time Scheduler
-  IEEE 802.11 Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications, Institute of Electrical and Electronics Engineers Std. 2012, 03 2012.
-  EN 302 665 Intelligent Transport Systems (ITS); Communications Architecture, European Telecommunications Standards Institute Std., Rev. V1,1,1, 09 2010.
-  IEEE 1609.0-2013 IEEE Guide for Wireless Access in Vehicular Environments (WAVE) - Architecture, IEEE Std., 2014.
-  T. L. Willke, P. Tientrakool, and N. F. Maxemchuk, “A survey of inter-vehicle communication protocols and their applications,” IEEE Communications Surveys Tutorials, vol. 11, no. 2, pp. 3–20, Second 2009.
-  R. Riebl, H. J. Günther, C. Facchi, and L. Wolf, “Artery: Extending veins for vanet applications,” in 2015 International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS), June 2015, pp. 450–456.
-  C. Sommer, R. German, and F. Dressler, “Bidirectionally coupled network and road traffic simulation for improved ivc analysis,” IEEE Transactions on Mobile Computing, vol. 10, no. 1, pp. 3–15, Jan 2011.
-  C. Obermaier, R. Riebl, and C. Facchi, “Dynamic scenario control for vanet simulations,” in Proceedings of IEEE MT-ITS 2017, 2017, unpublished.
-  K. C. Lee, S. h. Lee, R. Cheung, U. Lee, and M. Gerla, “First experience with cartorrent in a real vehicular ad hoc network testbed,” in 2007 Mobile Networking for Vehicular Environments, May 2007, pp. 109–114.
-  A. R. Khan, S. M. Bilal, and M. Othman, “A performance comparison of open source network simulators for wireless networks,” in 2012 IEEE International Conference on Control System, Computing and Engineering, Nov 2012, pp. 34–38.
-  G. Nirav and K. M. Sivalingam, “Dynamic routing framework for omnet++ based hardware-in-the-loop (hitl) network simulation,” in 2014 Twentieth National Conference on Communications (NCC), Feb 2014, pp. 1–6.
-  S. Boehm and M. Kirsche, “Looking into hardware-in-the-loop coupling of omnet++ and rosenet,” in Proceedings of the "OMNeT++" Community Summit 2015, 2015.
-  A. A. Scussel, G. Panholzer, C. Brandauer, and F. von Tüllenburg, “Improvements in omnet++/inet real-time scheduler for emulation mode,” in Proceedings of the “OMNeT++ Community Summit 2015”, 2015.
-  R. Riebl, C. Obermaier, S. Neumeier, and C. Facchi, “Vanetza: Boosting research on inter-vehicle communication,” in Proceedings of the 5th GI/ITG KuVS Fachgespräch Inter-Vehicle Communication (FG-IVC 2017), April 2017, pp. 37–40.
-  A. Wegener, M. Piórkowski, M. Raya, H. Hellbrück, S. Fischer, and J.-P. Hubaux, “TraCI,” in Proceedings of the 11th communications and networking simulation symposium on - CNS’08. ACM Press, 2008.
-  R. Riebl. (2017) Artery. [Online]. Available: https://github.com/riebl/artery
-  EN 302 637-2 Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Part 2: Specification of Cooperative Awareness Basic Service, European Telecommunications Standards Institute Std., Rev. V1,3,2, 11 2014.
-  Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Part 3: Specifications of Decentralized Environmental Notification Basic Service, European Telecommunications Standards Institute Std., Rev. V1,3,2, 11 2014.
-  EN 302 636-4-1 Intelligent Transport Systems (ITS); Vehicular Communications; GeoNetworking; Part 4: Geographical addressing and forwarding for point-to-point and point-to-multipoint communications; Sub-part 1: Media-Independent Functionality, European Telecommunications Standards Institute Std., Rev. V1,2,1, 07 2014.
-  R. Riebl, S. Neumeier, and C. Facchi, “Inter-vehicle communication on the run - experiences from tweaking veins runtime performance,” in Proceeding of the 3rd GI/ITG KuVS Fachgespräch Inter-Vehicle Communication. Ulmer Informatik-Berichte, 2015.
-  T. Biehle and K. Krumbiegel, Triggering Conditions and Data Quality - Dangerous Situation, CAR 2 CAR Communication Consortium Std., Rev. 3.3.0, 2015, release 1.1.0.
-  EN 302 663 Intelligent Transport Systems (ITS); Access layer specification for Intelligent Transport Systems operating in the 5 GHz frequency band, European Telecommunications Standards Institute Std., Rev. V1,2,1, 07 2013.