Identification of Model Uncertainty via Optimal Design of Experiments applied to a mechanical press
Abstract.
In engineering applications almost all processes are described with the aid of models. Especially forming machines heavily rely on mathematical models for control and condition monitoring. Inaccuracies during the modeling, manufacturing and assembly of these machines induce model uncertainty which impairs the controller’s performance. In this paper we propose an approach to identify model uncertainty using parameter identification and optimal design of experiments. The experimental setup is characterized by optimal sensor positions such that specific model parameters can be determined with minimal variance. This allows for the computation of confidence regions, in which the real parameters or the parameter estimates from different test sets have to lie. We claim that inconsistencies in the estimated parameter values, considering their approximated confidence ellipsoids as well, cannot be explained by data or parameter uncertainty but are indicators of model uncertainty. The proposed method is demonstrated using a component of the 3D Servo Press, a multitechnology forming machine that combines spindles with eccentric servo drives.
Key words and phrases:
model uncertainty, model inadequacy, optimal design of experiments, parameter identification, sensor placement, forming machinesCorresponding author]matei@mathematik.tudarmstadt.de
1. Introduction
In science, technology and economics mathematical models are commonly used to describe physical phenomena, to solve design problems and to manage production processes. The employment of these models entails various uncertainties. It has been observed that the dominant uncertainties arise from our lack of knowledge about system parameters and from deficiencies in the modeling itself [oden2010computer]. We understand models to be mathematical constructs consisting of variables, constants and their relations to each other. All present knowledge about the system or phenomenon of interest is described and represented by that model. We mean by model uncertainty that certain relations between model variables and parameters are simplified or insufficient to mirror reality. Thus, the present description of the system or phenomenon of interest is inaccurate or incomplete in the sense that there are aspects which have been ignored. As a consequence, any simulated process or manufactured product that is based upon these models is impaired in its predictive quality or usage. Hence, it is important to develop tools and algorithms for the identification and control of model uncertainty.
In order to identify whether a model is inadequate, one has to compare the model output to actual experimental data. It is, however, difficult to derive a simple criterion for the model to be accurate, since the measurement data is imperfect and subject to uncertainties as well. These data inaccuracies arise from irreducible randomness and from systematic errors in the measurement process. As a consequence, uncertainties are transferred from experimental data to the parameters. In the process of model calibration, the model parameters are adjusted such as to make them compatible to experimental observations. This is typically performed by solving inverse problems that are often illposed or underdetermined. Thus, additional prior information is required which is well known to be another source of uncertainty.
In this paper we propose an algorithm to detect model uncertainty using both parameter estimation and the optimal design of experiments approach. Here, we define parameter estimation to be the process of adjusting model parameters to the data and optimal design of experiments to be the best choice among experimental setups with reference to the reliability of the parameter estimates. Our methodology is able to distinguish between data and parameter uncertainties on the one hand and model uncertainty on the other hand. Particularly, we interpret inconsistencies in parameter estimates from different measurement series as an indicator that the underlying mathematical model is unable to describe all measurement series with the same set of parameter values. We assume neither an a priori distribution nor a specific form of model uncertainty in the mathematical equations.
The first step before estimating model parameters is to acquire measurements that capture the behavior of the system well. This step can be costly if many physical properties of the system have to be observed in each experiment. In certain engineering applications, the situation occurs that measurements are taken from a smallsized prototype instead of the expensive product which is often unavailable yet. It is therefore desirable to know beforehand the optimal sensor positions for the actual product by considering the results of experiments on the prototype. Thus, it is valuable to reduce the number of sensors if this does not downgrade the reliability of the parameter identification. Additionally, removing unreliable sensors may even improve the quality of the estimate. This can be done using the concept of optimal design of experiments, i.e., by deciding which sensors are actually best for gathering data in order to solve the parameter identification problem with minimal variance. Using these kinds of sensors and their optimal positions, measurements with optimal informational value can be obtained.
To determine model uncertainty based on these measurements, there are two possible procedures which depend on the previous knowledge regarding the model parameters. In the first case the real model parameters or good approximations thereof are known beforehand. In this case one can compute a confidence ellipsoid for a given confidence level , with a fixed value , around the parameters estimated from data and check whether the real parameter values (from prior knowledge) are contained in this ellipsoid. Unfortunately, in reality the actual values of the parameters are not always known exactly. Therefore, the second approach circumvents these difficulties in the following way: instead of using all measurements for the parameter identification problem, we split the experimental data into a calibration and a validation set. The approach starts by solving the parameter identification problem for the calibration set. Then we compute a confidence ellipsoid for a given confidence level and if the model is correct, then the solution of the parameter identification problem for the validation set should lie within this confidence ellipsoid. If the optimal parameters for the validation set are outside the confidence ellipsoid then this is an indication of model uncertainty.
In the literature, model uncertainty is especially discussed in modelbased fault diagnosis of machines. Simani et al. [simani2003model] treat uncertainties in the modeling by bounded error terms in the model equations. This method assumes a prior form of uncertainty which we do not.
Our approach is similar to the idea of Körkel et al. [koerkel2004], Bauer et al. [bauer2000oed] and Galvanin et al. [galvanin2007model] who also combined optimal design of experiments with parameter identification. However, they only used this method to reliably find optimal parameter values. Asprey and Macchietto [asprey2000statistical] continued this methodology to choose between competing models via maximizing a measure of the divergence of model predictions. In our approach no such measure is needed, we only employ the parameter estimates and their covariance matrices. Another difference is that we also consider higher order derivatives in the computation of the covariance matrix which is used to determine the confidence ellipsoid, see Bard [bard1974nonlinear] for more details.
There is extensive literature on Bayesian parameter calibration and validation. However, there seems to be only a few references dealing with model uncertainty from a general viewpoint. Lima et al. [lima2017selection] describe a general method to select the best model based on Occam’s Plausibility Algorithm [oden2017predictive] and Bayesian calibration. However, we do not adopt a Bayesian perspective but we involve design of experiments instead to sharpen the parameter estimates.
Staying within this Bayesian framework the same question whether a set of measurements for a given model is adequately described by the same set of parameters is addressed by Tuomi et al. [tuomi2011application]. Using a given prior distribution for the parameters, they derive an inequality to dismiss the assumption in case the probability for the data to be obtained under different parameter sets is significantly higher. In this work we discuss the same question but from a probabilistic point of view without any assumptions on the prior distribution of the parameters.
Another important approach to identify and control model uncertainty was created by Kennedy and O’Hagan [KennedyOHagan2001]. This method is based on the assumption that the true values of the quantities of interest are the sum of the model output , with input and parameters , and the model discrepancy term , i.e., the measurements should satisfy
(1) 
with independent observational noise . Then parameter identification can be performed for (1) to obtain best guesses for both the model parameters as well as the parameters of the model uncertainty . Arendt et al. [arendt2012quantification] use this approach for model updating and to distinguish between the effects of model calibration and model discrepancy. However, it has been shown by Brynjarsdóttir and O’Hagan [BrynjarsdottirOHagan2014] that the success of this approach heavily depends on incorporating a priori knowledge of the specific form of model uncertainty into the representation of , which is often assumed to be a specific type of stochastic process, but is actually not known beforehand. In contrast, our approach does not need any assumptions about the specific form of the model uncertainty.
One particular case of technical systems with a multitude of uncertain parameters and uncertain physical effects that challenge the modeling process are forming presses. Forming presses are highly loaded machines, which have kinematic degrees of freedom to perform a motion and to apply large forces on a workpiece. During that motion, the workpiece is then formed into a new shape. This can cause a considerable deflection of machine components which is of high technical importance and therefore we aim to model. In this application, we consider a mechanical forming machine which consists of a linkage mechanism. For this cause, the kinematic chain consisting of multiple mechanical components has to be approximated by a lumped parameter system to reduce the number of parameters. When modeling a machine we typically pursue one of two objectives that lead to different lumped parameter models: an accurate elastic behavior at low frequencies or an accurate frequency response [dresig14schwingungen]. In this application we seek a model that represents an accurate elastic behavior at low excitation frequencies. To estimate the stiffness of components with nonuniform crosssections, a finite element model is a typical technique. In a second step, the finite element model is reduced to the lumped parameter model. This model order reduction entails uncertainty besides a variety of uncertain influencing variables like material properties and inexact geometries. Hence, for some components it is necessary to identify the stiffnesses after the assembly of the machine. Due to the deflection, a relative movement of the components occurs and as a result friction dissipates a portion of this kinetic energy. However, for the modeling of friction on a macroscopic level, multiple phenomenological models exist so far [Bertotti.2006].
The paper is organized as follows. First we introduce the parameter identification problem and its covariance estimation. Based on the resulting covariance matrix, we then formulate the problem of optimal experimental design to find optimal sensor positions which lead to the smallest variance of the resulting parameter identification. In Section 4 we describe in more detail how parameter identification and optimal design of experiments can be used to identify and control model uncertainty. Afterwards we introduce the working principle and the mathematical model of the 3D Servo Press as a practical example. The application of our proposed method to the 3D Servo Press is done in Section 6, where we also present numerical results. We end the paper by giving some concluding remarks.
2. The Parameter Identification Problem and its Covariance Estimation
In this section we present the parameter identification problem as it is done by Körkel et al. [koerkel2004]. We first introduce some basic notation and assumptions, formulate the parameter estimation problem and then deduce the covariance matrix as well as the considered confidence regions.
The mathematical model is given by a state equation coupling the state vector and the parameters for any control variable , i.e.,
(2) 
This state equation may be a discretized form of a partial differential equation with large dimensions and . We assume that (2) has a unique solution for any given and .
The model parameters are in general not known beforehand. Therefore, we need measurement data to obtain appropriate estimates. Let denote the number of allocated sensors for data collection and be the total number of performed measurement series. The control variables represent external boundary forces in our case. Now, the aim of the parameter identification problem is to find model parameters that best fit the measurements for given control . We assume that the measurements are collected by prepositioned sensors where each sensor is accurate up to a standard deviation for each control and in each measurement series . In general, it is not possible to measure all of the state components directly. Therefore, we introduce an observation operator that maps state, parameters and controls to the actual quantity that is measured. Since we will later choose an optimal subset of all the possible sensors, we introduce binary weights such that if and only if sensor is used.
We now apply the leastsquares method to find the optimal parameter values which minimize the discrepancy between measurement data and model output weighted by the standard deviation of each sensor, respectively.
(3)  
Remark 1.
Alternatively, we can also assume that each sensor has a given standard deviation in each measurement scenario and for each control . However, to keep notation simple, we assume the working precision of each sensor to be constant over all measurement series and all controls.
For convenience we rewrite problem (3) in vector form of dimension and eliminate the state equation by inserting the unique state solution
into the objective function leading to the optimization problem
(4) 
with the notations
Thus, the vector consists of copies of the vector , while and are diagonal matrices consisting of copies of and , respectively. The measurement tensor is vectorized as well and for convenience we use the same symbol.
The inverse problem (3) or rather (4) can be solved using, e.g., an extended GaussNewton method, see Dennis et al. [Dennis1981] for more details. We denote the solution of this optimization problem by .
For the quantification of data uncertainty we now assume the measurement errors to be normally and independently distributed, i.e.,
where are the true but unknown values of the quantities that are measured. Since the measurement data is a random variable, the estimated parameters are also random variables. Thus, we are interested in how a perturbation of propagates to . Since is unknown in general, we take some given measurements as a best known approximation. To compute a distribution of , we linearize the solution operator of the parameter identification problem around , such that the linearized parameter identification is again Gaussian distributed, compare, e.g., Proposition 3.2 in [Ross2010]. In order to characterize this distribution, we compute the covariance matrix for the linearized parameter estimation, while we approximate the mean by the solution of the nonlinear parameter estimation problem (4) to the given set of measurements . For convenience to the reader we write instead of . The covariance matrix is then given by
(5)  
where we approximate the mean by .
Thus, the approximated confidence ellipsoid for a certain level of the multivariate Gaussian distributed solution of the parameter identification is given by
(6) 
where is the quantile of the distribution with degrees of freedom. For more details on multivariate Gaussian distributions and confidence ellipsoids, see for example Scheffé [Scheffe1960].
To derive an analytical expression of the covariance matrix in (5), following Bard [bard1974nonlinear], we use the linearized version of the mapping at :
The sensitivity can then be determined using the first order optimality condition for the parameter identification problem (4), i.e,
(7) 
In order to use the implicit function theorem, we make the following assumption:
Assumption 1.

is twice continuously differentiable with respect to .

is invertible.
Remark 2.
Note that Assumption 1 (i) is implied by the condition that the observation operator is twice continuously differentiable with respect to .
Using Assumption 1, we can now apply the implicit function theorem. Thus, equation (7) implicitly defines a mapping and its derivative is given by
(8) 
in any direction . More precisely, we have
Let us define
with and
where and are matrices. Thus, . The exact calculation of and is given in the appendix, which requires the following assumption to allow the usage of the implicit function theorem:
Assumption 2.

The state equation is twice continuously differentiable.

is invertible.
We want to make sure that the principal part stays invertible when perturbing the sensor variables:
Assumption 3.
The matrix has full column rank, i.e., , and .
From this assumption we can infer invertibility of , compare Körkel et al. [koerkel2004] for more details. In other words it is always necessary to employ at least as many sensors as the number of parameters which should be estimated. This is an important constraint later in the optimization with respect to the sensor variables.
Thus, from (8) we obtain
Using the calculations from above, the approximated covariance matrix is given by
(9)  
3. Optimal Design of Experiments
The optimal design of experiments problem deals with the task of finding optimal sensor positions such that the reliability of the parameter estimation is maximized. The reliability also depends on the accuracy of the sensors used for the measurements, whereby each sensor has a given variance .
It is very common to measure the reliability of the parameter estimation by a singlevalued design function , see Bauer et al. [bauer2000oed] and Franceschini and Macchietto [franceschini2008model]. It is obvious that a small covariance leads to a high reliability of the parameter estimation. However, it is unclear what a small covariance means in terms of matrices. In general, there are different approaches how to choose the function. We list the most prominent ones according to Fedorov and Leonov [fedorov2013optimal]

Acriterion: , the square of the halflength of the rectangle’s diagonal that circumscribes the confidence ellipsoid,

Dcriterion: , the square of the product of halflengths of the confidence ellipsoid’s principal axes,

Ecriterion: , the square of the halflength of the largest principal axis.
It seems natural to use the Dcriterion due to its close connection to the volume of the confidence ellipsoid and its invariance with respect to transformations applied to the model parameters. However, this criterion tends to emphasize the most sensitive parameter [franceschini2008model]. The Acriterion ignores the amount of information on the offdiagonal elements of the covariance matrix. This is particularly inefficient when there is a high correlation between parameters. For the numerical example in this paper, we choose the Ecriterion even though Eoptimality may lead to a tolerable increase in volume of the confidence ellipsoid. From an experimental point of view, the ideal confidence region would be a rectangle. This enables experimenters to give confidence intervals for the quantities of interest. The Ecriterion may serve this purpose by reducing the largest expansion of the confidence ellipsoid.
We formulate the optimal design of experiments problem as follows:
(10)  
The constraint is a general, possibly nonlinear function which describes for example bounds on the number of used sensors. In our case, to fulfill the rank condition in Assumption 3, must contain the inequality .
The optimal design of experiments problem (10) is thus a nonconvex mixedinteger nonlinear program (MINLP). Such problems can be solved via spatial branchandbound, see, e.g., Burer and Letchford [BurerLetchford2012] for an overview.
Note, however, that for the correctness of the proposed approach, problem (10) does not necessarily need to be solved to optimality. Using a good but suboptimal sensor placement will not lead to any incorrect rejection of a model, since the variance of the parameter estimation becomes larger and therefore also the confidence ellipsoids increase. Thus, it is also possible to solve (10), which is the computationally most expensive step of the proposed approach, by heuristic methods. In our numerical example, the number of sensors is very small, so that a heuristic method may indeed provide satisfiable results.
4. Detecting Model Uncertainty
In this section, we discuss how optimal design of experiments and parameter identification can be used to detect model uncertainty. To do so, assume that all parameters of the model have a true physical meaning and that in case the model is correct, the solution of the parameter identification problem, computed for a sufficiently large number of measurement series with zero noise, corresponds to these real parameters. Then repeated solutions of the parameter identification problem for different measurements with differing controls should, within the boundaries of the model up to uncertainty of the measurements, deliver the same set of parameters. On the other hand, if one set of measurements leads to parameters which lie outside a given confidence set of the previous runs, then this implies that the model cannot replicate the results of all measurements reliably, i.e., the underlying model is inadequate.
As already explained in the introduction, the first step of identifying parameters of the model by fitting them to a given set of measurements is to actually acquire these measurements which can be extremely costly. Furthermore, the quality of the parameter estimation may even be improved by removing unreliable sensors. Therefore, we only acquire a minimal amount of measurement series which is needed for the computation of the optimal design of experiments introduced in the previous section to determine optimal sensor positions. In this case, we solve problem (10) with a restriction on the desired minimal number of used sensors to decide which sensors are actually essential to solve the parameter identification problem with minimal variance.
Our approach to detect model uncertainty is depicted in Figure 1. If the parameter values are known by a prior information source, e.g., by previously performed experiments, then we directly solve the optimal design of experiments problem to generate additional measurements with minimal costs. New parameter values are estimated from these data sets together with their confidence ellipsoid. If the previously known parameter values are outside this confidence ellipsoid then this is an indication of model uncertainty and the loop starts again at the deduction of a mathematical model.
Now assume that a sufficiently large set of measurements for variance minimal sensors is given and assume that the real values of the parameters which should be estimated are not known beforehand. Then split the test set into a calibration set and a validation set . This split can either be done in a homogeneous way to test whether the model is accurate in general, or it can be chosen in a way to test whether a specific physical effect is sufficiently modeled. For example, the test set could be split according to the magnitude of the controls to check if the results for both sets can be reproduced by the model for the same set of parameters. On the one hand, this approach can help to identify ranges of control variables for which the model works better or worse, and on the other hand, to detect specific effects which are not yet sufficiently implemented in the model.
The main part of the algorithm starts by solving the parameter identification problem for the calibration set leading to a solution . From the solution we can then compute the corresponding covariance matrix of the parameter identification as given in (9). This covariance matrix allows us to compute a confidence ellipsoid for a given confidence level using Equation (6). If the model is correct, then the solution of the parameter identification problem for the validation set should lie within this confidence ellipsoid. If the optimal parameters for the validation set are outside the confidence ellipsoid, this is an indication of model uncertainty and the loop starts again at the deduction of a mathematical model, see Figure 1.
5. The 3D Servo Press Model
The method for detecting uncertainty is demonstrated at hand of a technical system, the 3D Servo Press [scheitza10konzeption], a forming machine which transmits the torques and forces of its drives onto a part to be formed, e.g. a car body part. Therefore, a forming machine is subject to high external forces during its motion which cause its mechanism to deflect. While a rigid body model is accurate during the unloaded state, it does not suffice during the forming operation [groche17stiffness]. Especially for the closedloop control of forming machines, an accurate model is crucial as inaccuracies can cause the control to become unstable [hoppe19closedloop]. However, the modeling of forming machines requires a high degree of abstraction, since elastic bodies need to be reduced to bars and beams. Furthermore, nonlinear bearing stiffnesses as well as friction have to be taken into account.
Figure 2 shows the 3D Servo Press that consists of three identical linkage mechanisms. We use a mechanical substitute model and describe it for one linkage mechanism. A variety of bars and beams are connected via joints that are designed as rotary joints. Each elastic component is represented by a spring or beam and each mass by a gray volume. The eccentric and spindle drives move the three degrees of freedom of one gear unit , that cause all joints in the kinematic chain to perform a desired movement. The output of the gear unit is point , which leads down to the ram bearing via a linear pressure bar. For the rigidbody model, the position of all points is defined by the angle of the eccentric drive as well as the upper and lower spindle drive position .
To model the elastic 3D Servo Press, the coupling links are interpreted as bars and beams, depending on their stress state under load. The bar and beam models are composed of masses and springs. The bearings are modeled as simple spring elements with either linear or nonlinear spring characteristics. The equation of motion of the system is drawn up using the Lagrange equations of the second kind
(11) 
where is the Lagrangian consisting of the total kinetic energy and the total potential energy , are the system states and are the nonconservative forces. The nonconservative forces contain all external forces that are applied to the machine, i.e., the torque of the eccentric drive , the forces of the upper and lower spindles , and the reacting process force . In this application we want to evaluate the elastic model and therefore fix the drives positions. Thus, only is applied and all other nonconservative forces are zero.
Solving the Lagrangian equation requires the potential and kinetic energy as a function of the states. These consist of the stored energy in each elastic and rigid body
whereby the energies of the individual elements are given as follows.

[listparindent=]

Bar model
A direct approach to discretizing the bar while maintaining inertia and rigidity is the finite element method. It is based on the partial differential equation of the continuous bar and supplies the mass matrix and the stiffness matrix for an element of mass and stiffness , which are given by
As the actual elements do not have a uniform cross section, the stiffness is determined using a finite element simulation based on the ideal CAD model.
Remark 3.
However, the CAD model and finite element model are based on the detailed knowledge of the elastic modulus and the geometry of the components. Due to natural fluctuations in material production, the elastic modulus may vary from part to part. In addition, manufacturing limitations only impede geometric accuracy. Therefore, determining the stiffness by an a priori FEM simulation leads to an uncertain estimation of the actual stiffness and requires a parameter identification based on posterior measurements.
The kinetic energy of an individual bar shown in Figure 3 sums up to
with the translational velocities of the masses , the mass moment of inertia and the corresponding rotational velocity . Its potential energy originates from the energy stored in the elasticity and the gravitational potential energy of the masses
where is the elongation of the element, is the standard gravity of Earth and is the relative distance of each mass to the ground.

Beam model
All elements that experience bending moments are modeled as beams. This applies especially to the lever, which connects three points instead of two and is marked as a solid line in Figure 2. Like the bar model, the beam model is based on the equations of the finite element method and serves as the basis for modeling the lever under bending load. Since the lever in total features three joints, the model can be seen as two flat beam elements arranged in a row. A lumped mass model is set up in which all elements outside the main diagonal of the mass matrix are neglected. The stiffness of each finite element results in a stiffness matrix
using the simulated stiffnesses , and the length of the beam . Since the lever consists of two finite elements, two element matrices are joined together to form a stiffness matrix according to the finite element method. The result is the stiffness matrix . The total mass of the lever is distributed to the model masses
As the kinetic energy of a beam shown in Figure 4 is equivalent to the kinetic energy of a bar, this results in
where is the rotation of the complete beam. For the calculation of the potential energy, the sum of the positional energy of the masses and the elastic energy
is calculated where
are the states of the beam.

Bearing model
The bearings are modeled as spring elements between the joints of the couplers. Since the radial bearing force applied by the bearings is a function of deflection, the deflection must be described with the position coordinates of the bodies. Assuming a constant joint stiffness, the potential energy results in
with the joint’s stiffness and its radial deflection .

Friction model
Friction occurs in all bearings in which a relative movement takes place and will cause a hysteresis in the loaddisplacement curve. As the relative movements in the joints is small compared to the movement of the pressure bar that connects point with point (see Figure 2), only the bearings guiding that bar are considered. Nevertheless, a variety of model approaches exist for friction so far. In order to test which approach is the closest to reality in this case, three rateindependent friction models of different complexity are pursued.

[label=0)]

Since friction is hard to model, it is often neglected which leads to the model

The discontinuous Coulomb friction model
(12) gives a more accurate description of friction in which is a friction constant. As we can assume that the sign of is the same as the sign of we can simplify the model to be only discontinuous in the input variables and not in the states.

As a third model approach, a continuous friction model with rateindependent memory that takes into account past force data is considered [Bertotti.2006]. Here, we take into account the force of the current and last time step
as well as the minimum and maximum force value during loading and unloading cycles
that are internal variables and reduce the complexity of memorizing an infinite number of time steps . Based on the Preisach model [Preisach.1935] which is a discontinuous hysteresis model, we used an adapted continuous model which is comparable to a neural network topology [Mayergoyz.2003]. Figure 5 shows the topology of the used model where .
To train the model, we have to determine the friction force which is the difference of the actual measured process force and estimated force by the inverse model. The inverse model describes the required force under a measured displacement and contains the estimated stiffness parameters that have been determined without any friction model in a first step. Applying this to all 29 measurements of a loading cycle, the full hysteresis can be identified and used to train the friction model.

Synthesis of the press model
The press model consists of 2 rigid bodies, 5 bars, 1 beam, 10 joints and the elasticity of the press frame which represents support points to the environment. This results in a 34dimensional state vector .
Equation (11) can now be written as
with the mass matrix , the contribution of the kinetic energy and of the potential energy and the excitation forces
In this case we are interested in the quasistatic model to identify uncertain stiffness parameters of two bars and , in the following denoted as and as shown in Figure 2. Thus, all derivatives of are set to zero such that
where the function contains the parameters and .
To identify the model parameters, a process force is applied using an external pneumatic force source.
6. Numerical Results for the 3D Servo Press
We implemented the described procedure to detect model uncertainty using MATLAB R2017b and the included lsqnonlin solver for the parameter identification problems and applied it to the gear mechanism model of the 3D Servo Press.
We use measurements for different process forces (these are the control variables), whereby the first forces describe loading and the last describe unloading of the 3D Servo Press. For each force variable we measure the vertical displacements in point , the horizontal displacements in point and the vertical displacements in point when applying a vertical process load on the press, see Figure 2. The displacements are measured in and the forces in . In this section we adopt again the notation from Section 2.
Each measurement is performed times although with slightly differing forces due to variations in the pneumatic pressure when applying the force.
In a first step, we analyze the measurement data. In our modeling we assumed that the measurements are normally distributed. Since the true values of the quantities that are measured are unknown to us, we check whether the measurement errors are normally distributed instead. In order to verify this assumption, we first perform a chisquared goodnessoffittest [Greenwood1996] applied to the measurement errors with confidence . To do so, we calculate the standard deviation of each sensor using all the 116 measurements of the random variables
If are independent and identically distributed with the same mean and standard deviation then we can easily see that each row in is independent and identically distributed with mean zero and
and therefore
for and , respectively, and in the same way for the last row of . Now we test the hypotheses that each is normally distributed with mean zero and variance estimated from for each sensor . The results of the hypothesis tests are shown in Table 2:
Sensor  Hypotheses  value (in %)  Sigma 

1  0  13.78  9.6990e06 
2  0  17.21  8.9183e06 
3  1  0.11  3.2217e06 
We observe that the hypotheses should be accepted for sensors 1 and 2. For sensor 3 the hypotheses is formally rejected. However, one has to consider that in each experiment the magnitudes of the applied load forces could not be kept constant, they differ slightly in each measurement series. This is an unavoidable limiting factor in our experimental setup. Therefore, one cannot conclude that sensor 3 captures nonGaussian experimental data. We assume nevertheless that all sensors return data that is normally and independently distributed.
As explained above, the 29 control forces could not be kept constant in all measurement series. However, the deviations are small and the computed standard deviations for the sensors include this effect. Thus, it is justifiable to work with the means from now on:
Having the initial experimental data available, the aim is now to reduce the costs for obtaining these measurements, i.e., we want to reduce the number of involved sensors. We first restrict the set of parameters to and shown in Figure 2, while the remaining model parameters are fixed to values computed by a past parameter identification. The parameters and describe the axial stiffness of elastic components of the 3D Servo Press, see Section 5. Since the number of involved sensors must be greater or equal to the number of estimated parameters, compare Assumption 3, we want to choose two of the three sensors which together allow the most reliable parameter identification with smallest value for the Ecriterion.
For comparison, we compute all three design criteria that are mentioned in Section 3 using optimal parameters obtained by the solution of the parameter identification problem. We see the following result:
Sensor combination  

111 (initial)  9.2205e+09  4.1828e+16  9.2160e+09 
101  1.1363e+29  7.7269e+35  1.1363e+29 
011  9.4774e+09  4.7946e+16  9.4723e+09 
110  3.4949e+09  6.5704e+16  3.4760e+09 
We observe that omitting the second sensor increases all design criteria by a factor of compared to the initial sensor configuration, which is an indication that the covariance matrix became close to singular. A removal of the first sensor, though, increases the maximal eigenvalue and the volume of the confidence ellipsoid slightly. However, omitting the last sensor, i.e., measuring the vertical displacements in point , leads to the smallest maximal eigenvalue. We choose the Ecriterion as design criterion for reasons explained in Section 3. Thus, we proceed with the optimal sensor combination , i.e., we choose to measure the vertical displacements in point and the horizontal displacements in point .
The experiments revealed a hysteresis behavior. This has a significant contribution to model uncertainty. We want to test whether our algorithm recognizes the best out of three different models used to describe the data. Therefore, we introduce the following models:
Compare Section 5 for a description of the last two models. Figure LABEL:fig:meas3DSP shows the different behavior of these models plotted together with the data.
For the assembly of we need real measurements to train the neural network. For this purpose we take three data series away from our available amount of data such that only measurement series remain for the application of our algorithm to detect model uncertainty. In order to make the following test strategy fair, we use only these four measurement series for all models alike since the more data series are involved the harder it is for the model to reproduce them all.
The standard deviation of the sensors is crucial for the size of the confidence ellipsoid of the parameter estimates. We fix these values to be the sum of the standard deviation of the repeated measurement process (repetition accuracy) plus the standard deviation of the linearization error and other internal errors as specified by the manufacturer of the sensor. Thus, we take the values 1.890e05, 9.241e06 and 5.184e06 for the total accuracy of the sensors.
In order to investigate the validity of the models and , we generate homogeneous calibration and validation sets. We first split the test set consisting of the four measurement series into a loading and unloading test set and consider each set separately. Thus, for the loading set , we again split the test set homogeneously into a calibration and a validation test set. Next, we test loading versus unloading and again split the set homogeneously for calibration and validation . Lastly, we test loading together with unloading and split the set homogeneously for calibration and validation , compare Table 3.
Force progression  Calibration  Validation 

Loading  
Unloading  
Loading vs. Unloading  
Loading and Unloading 
Now, for each of the three models and for each of the four test scenarios we try the hypotheses whether the estimated parameters from the corresponding validation data set lie inside the confidence ellipsoid around the parameter estimate of the corresponding calibration data set. Table 4 lists the results. The last three columns show the minimal confidence level, i.e., the value of statistics, such that the parameters of the corresponding validation test set lie on the boundary of the confidence ellipsoid of the corresponding calibration test set.
Calibration  Validation  value (in %)  value (in %)  value (in %) 

for  for  for  
0.53  76.92  99.52  
0  54.37  86.61  
0  0  7.57  
0.20  0  96.55 
For the usual choice of the confidence level , when comparing the values, we clearly see that the model , which does not account for hysteresis, is rejected for all test scenarios. Thus, we detected that is uncertain. The data cannot be described by a simple linear model. We demand to be updated such as to correctly represent hysteresis. This is done in a first attempt by the Coulomb friction model, see equation (12). We thus perform our algorithm on . While this model seems to be able to describe loading and unloading separately it fails to describe both scenarios with the same set of parameters. Since hysteresis is a continuous effect, the discontinuous Coulomb friction model still fails to reproduce the fine nuances of the experimental data. Our proposed method is able to identify this in the third and fourth test scenario, where the model is clearly rejected since the value is close to zero. Hence, a neural network has been employed to improve the model output as mentioned in Section 5. The last column of Table 4 shows that model is wellsuited to explain the hysteresis phenomenon.
To sum up we have seen that the method is able to detect model uncertainty and by suitable choice of the calibration and validation test sets, it can even help to identify particular aspects of the model that need to be improved.
7. Conclusion
In this paper we have seen how model uncertainty can be identified by combining the optimal design of experiments approach with parameter identification. Optimal design of experiments can be used to choose sensors which allow the parameter identification problem to be solved with minimal variance. Using the covariance matrix of the parameter estimate, we can then compute confidence ellipsoids which should include the parameter estimates with high probability. If some other test set leads to a solution of the parameter identification outside such a confidence ellipsoid, then we can conclude with high probability that not all of the measurements can be explained by the same model with the same set of parameters. We then introduced the 3D Servo Press as an application and demonstrated our approach on mathematical models of the press. This allowed us to show that two simple models were not valid, since specific effects like hysteresis are not sufficiently modeled. A sophisticated mirroring of the hysteresis effect led to a model for which no more model uncertainty could be detected. Thus, the goal of identifying model uncertainty for the 3D Servo Press was achieved.
It would be interesting to further test our method with models that depend on more than two parameters and to have a larger number of possible sensor locations available. Furthermore, instead of only choosing sensors once in the beginning, it would also be possible to resolve the optimal design of experiments problem using the parameters identified through some first experiments to iteratively strengthen the quality of the parameter identifications, in a similar way as proposed by Körkel et al. [koerkel2004].
8. Acknowledgement
This research was funded by the German Research Foundation (DFG) – project number 57157498 – CRC 805. The authors would like to thank the DFG for funding.
References
Appendix
To compute , we need to derive the first and second derivatives of with respect to . For convenience to the reader we write instead of :
(13)  
To determine the terms and , we again apply the implicit function theorem, using Assumption 2, yielding
(14) 
Moreover, the second directional derivatives of with respect to in directions and is given by
(15)  
The exact characterization of the vectortensor and matrixtensor products in equation (13) and (15) above is given by
whereby denotes the standard tensor product and with dropped dependencies on and for the sake of clarity. Altogether, , and therefore also can be determined using the expressions given in (13)–(15).