We present variational sequential optimal experimental design (vsOED), a novel method for optimally designing a finite sequence of experiments within a Bayesian framework with information-theoretic criteria. vsOED employs a one-point reward formulation with variational posterior approximations, providing a provable lower bound to the expected information gain. Numerical methods are developed following an actor-critic reinforcement learning approach, including derivation and estimation of variational and policy gradients to optimize the design policy, and posterior approximation using Gaussian mixture models and normalizing flows. vsOED accommodates nuisance parameters, implicit likelihoods, and multiple candidate models, while supporting flexible design criteria that can target designs for model discrimination, parameter inference, goal-oriented prediction, and their weighted combinations. We demonstrate vsOED across various engineering and science applications, illustrating its superior sample efficiency compared to existing sequential experimental design algorithms.

Most general experimental design algorithms are either: (i) stochastic and hence give different designs each time they are run with finite computing power, or (ii) deterministic but converge to results that depend on an initial or reference design, taking little or no account of the range of all other possible designs. In this paper we introduce an approximation to standard measures of experimental design quality that enables a new algorithm to be used. The algorithm is simple, deterministic and the resulting experimental design is influenced by the full range of possible designs, thus addressing problems (i) and (ii) above. Although the designs produced are not guaranteed to be globally optimal, they significantly increase the magnitude of small eigenvalues in the model–data relationship (without requiring that these eigenvalues be calculated). This reduces the model uncertainties expected post-experiment. We illustrate the method on simple tomographic and microseismic location examples with varying degrees of seismic attenuation

Nonlinear regression models addressing both efficacy and toxicity outcomes are increasingly used in dose-finding trials, such as in pharmaceutical drug development. However, research on related experimental design problems for corresponding active controlled trials is still scarce. In this paper we derive optimal designs to estimate efficacy and toxicity in an active controlled clinical dose finding trial when the bivariate continuous outcomes are modeled either by polynomials up to degree 2, the Michaelis- Menten model, the Emax model, or a combination thereof. We determine upper bounds on the number of different doses levels required for the optimal design and provide conditions under which the boundary points of the design space are included in the optimal design. We also provide an analytical description of the minimally supported $D$-optimal designs and show that they do not depend on the correlation between the bivariate outcomes. We illustrate the proposed methods with numerical examples and demonstrate the advantages of the $D$-optimal design for a trial, which has recently been considered in the literature.

Deriving optimal designs for nonlinear models is challenging in general. Although some recent results allow us to focus on a simple subclass of designs for most problems, deriving a specific optimal design mainly depends on algorithmic approaches. There is need of a general and efficient algorithm which is more broadly applicable than the current state of the art methods. We present a new algorithm that can be used to find optimal designs with respect to a broad class of optimality criteria, when the model parameters or functions thereof are of interest, and for both locally optimal and multi- stage design strategies. We prove convergence to the desired optimal design, and show that the new algorithm outperforms the best available algorithm in various example

In the context of estimating material properties of porous walls based on in-site measurements and identification method, this paper presents the concept of Optimal Experiment Design (OED). It aims at searching the best experimental conditions in terms of quantity and position of sensors and boundary conditions imposed to the material. These optimal conditions ensure to provide the maximum accuracy of the identification method and thus the estimated parameters. The search of the OED is done by using the Fisher information matrix and a priori knowledge of the parameters. The methodology is applied for two case studies. The first one deals with purely conductive heat transfer. The concept of optimal experiment design is detailed and verified with 100 inverse problems for different experiment designs. The second case study combines a strong coupling between heat and moisture transfer through a porous building material. The methodology presented is based on a scientific formalism for efficient planning of experimental work that can be extended to the optimal design of experiments related to other problems in thermal and fluid sciences.

Questions of ‘how best to acquire data’ are essential to modelling and prediction in the natural and social sciences, engineering applications, and beyond. Optimal experimental design (OED) formalizes these questions and creates computational methods to answer them. This article presents a systematic survey of modern OED, from its foundations in classical design theory to current research involving OED for complex models. We begin by reviewing criteria used to formulate an OED problem and thus to encode the goal of performing an experiment. We emphasize the flexibility of the Bayesian and decision-theoretic approach, which encompasses information-based criteria that are well-suited to nonlinear and non-Gaussian statistical models. We then discuss methods for estimating or bounding the values of these design criteria; this endeavour can be quite challenging due to strong nonlinearities, high parameter dimension, large per-sample costs, or settings where the model is implicit. A complementary set of computational issues involves optimization methods used to find a design; we discuss such methods in the discrete (combinatorial) setting of observation selection and in settings where an exact design can be continuously parametrized. Finally we present emerging methods for sequential OED that build non-myopic design policies, rather than explicit designs; these methods naturally adapt to the outcomes of past experiments in proposing new experiments, while seeking coordination among all experiments to be performed. Throughout, we highlight important open questions and challenges.

The scope of conjoint experiments on which we focus embraces those experiments in which each of the respondents receives a different set of profiles to rate. Carefully designing these experiments involves determining how many and which profiles each respondent has to rate and how many respondents are needed. To that end, the set of profiles offered to a respondent is viewed as a separate block in the design and a respondent effect is incorporated in the model, representing the fact that profile ratings from the same respondent are correlated. Optimal conjoint designs are then obtained by means of an adapted version of the algorithm of Goos and Vandebroek (2004). For various instances, we compute the optimal conjoint designs and provide some practical recommendations.Conjoint analysis; D-Optimality; Design; Model; Optimal; Optimal block design; Rating-based conjoint experiments; Recommendations;

The real-world testing of decisions made using causal machine learning models is an essential prerequisite for their successful application. We focus on evaluating and improving contextual treatment assignment decisions: these are personalised treatments applied to e.g. customers, each with their own contextual information, with the aim of maximising a reward. In this paper we introduce a model-agnostic framework for gathering data to evaluate and improve contextual decision making through Bayesian Experimental Design. Specifically, our method is used for the data-efficient evaluation of the regret of past treatment assignments. Unlike approaches such as A/B testing, our method avoids assigning treatments that are known to be highly sub-optimal, whilst engaging in some exploration to gather pertinent information. We achieve this by introducing an information-based design objective, which we optimise end-to-end. Our method applies to discrete and continuous treatments. Comparing our information-theoretic approach to baselines in several simulation studies demonstrates the superior performance of our proposed approach.

Bayesian optimal experimental design (OED) seeks experiments that maximize the expected information gain (EIG) in model parameters. Directly estimating the EIG using nested Monte Carlo is computationally expensive and requires an explicit likelihood. Variational OED (vOED), in contrast, estimates a lower bound of the EIG without likelihood evaluations by approximating the posterior distributions with variational forms, and then tightens the bound by optimizing its variational parameters. We introduce the use of normalizing flows (NFs) for representing variational distributions in vOED; we call this approach vOED-NFs. Specifically, we adopt NFs with a conditional invertible neural network architecture built from compositions of coupling layers, and enhanced with a summary network for data dimension reduction. We present Monte Carlo estimators to the lower bound along with gradient expressions to enable a gradient-based simultaneous optimization of the variational parameters and the design variables. The vOED-NFs algorithm is then validated in two benchmark problems, and demonstrated on a partial differential equation-governed application of cathodic electrophoretic deposition and an implicit likelihood case with stochastic modeling of aphid population. The findings suggest that a composition of 4--5 coupling layers is able to achieve lower EIG estimation bias, under a fixed budget of forward model runs, compared to previous approaches. The resulting NFs produce approximate posteriors that agree well with the true posteriors, able to capture non-Gaussian and multi-modal features effectively.

So far, the optimal design of blocked and split-plot experiments involving mixture components has received scant attention. In this paper, an easy method to construct efficient blocked mixture experiments in the presence of fixed and/or random blocks is presented. The method can be used when qualitative variables are involved in a mixture experiment as well. It is also shown that orthogonally blocked mixture experiments are highly inefficient compared to D-optimal designs. Finally, the design of a split-plot mixture experiment with process variables is discussed.Design; Fixed and random blocks; Minimum support design; Mixture experiment; Optimal; Optimal design; Orthogonal blocking; Process variables; Processes; Qualitative variables; Split-plot experiment; Variables;

Optimal Bayesian design techniques provide an estimate for the best parameters of an experiment in order to maximize the value of measurements prior to the actual collection of data. In other words, these techniques explore the space of possible observations and determine an experimental setup that produces maximum information about the system parameters on average. Generally, optimal Bayesian design formulations result in multiple high-dimensional integrals that are difficult to evaluate without incurring significant computational costs as each integration point corresponds to solving a coupled system of partial differential equations. In the present work, we propose a novel approach for development of polynomial chaos expansion (PCE) surrogate model for the design utility function. In particular, we demonstrate how the orthogonality of PCE basis polynomials can be utilized in order to replace the expensive integration over the space of possible observations by direct construction of PCE approximation for the expected information gain. This novel technique enables the derivation of a reasonable quality response surface for the targeted objective function with a computational budget comparable to several single-point evaluations. Therefore, the proposed technique reduces dramatically the overall cost of optimal Bayesian experimental design. We evaluate this alternative formulation utilizing PCE on few numerical test cases with various levels of complexity to illustrate the computational advantages of the proposed approach.

An optimal experimental set‐up maximizes the value of data for statistical inferences. The efficiency of strategies for finding optimal experimental set‐ups is particularly important for experiments that are time‐consuming or expensive to perform. In the situation when the experiments are modeled by partial differential equations (PDEs), multilevel methods have been proven to reduce the computational complexity of their single‐level counterparts when estimating expected values. For a setting where PDEs can model experiments, we propose two multilevel methods for estimating a popular criterion known as the expected information gain (EIG) in Bayesian optimal experimental design. We propose a multilevel double loop Monte Carlo, which is a multilevel strategy with double loop Monte Carlo, and a multilevel double loop stochastic collocation, which performs a high‐dimensional integration on sparse grids. For both methods, the Laplace approximation is used for importance sampling that significantly reduces the computational work of estimating inner expectations. The values of the method parameters are determined by minimizing the computational work, subject to satisfying the desired error tolerance. The efficiencies of the methods are demonstrated by estimating EIG for inference of the fiber orientation in composite laminate materials from an electrical impedance tomography experiment.

Finding optimal designs for experiments for non-linear models and dependent data is a challenging task. We show how the problem simplifies when the search is restricted to designs that are minimally supported; that is, the number of distinct runs (treatments) is equal to the number of unknown parameters, p, in the model. Under this restriction, the problem of finding a locally or pseudo-Bayesian D-optimal design decomposes into two simpler problems that are more widely studied. The first is that of finding a minimum- support D-optimal design d1 with p runs for the corresponding model for the mean but assuming independent observations. The second problem is finding a D-optimal block design for assigning the treatments in d1 to the experimental units. We find and assess optimal minimum-support designs for three examples, each assuming a mean model from a different member of the exponential family: binomial, Poisson and normal. In each case, the efficiencies of the designs are compared to the optimal design where the restriction on the number of distinct support points is relaxed. The optimal minimum-support designs are found to often perform satisfactorily under both local and Bayesian D-optimality for concentrated prior distributions. The results are also relatively insensitive to the assumed degree of dependence in the data

Consideration set formation using non-compensatory screening rules is a vital component of real purchasing decisions with decades of experimental validation. Marketers have recently developed statistical methods that can estimate quantitative choice models that include consideration set formation via non-compensatory screening rules. But is capturing consideration within models of choice important for design? This paper reports on a simulation study of a vehicle portfolio design when households screen over vehicle body style built to explore the importance of capturing consideration rules for optimal designers. We generate synthetic market share data, fit a variety of discrete choice models to the data, and then optimize design decisions using the estimated models. Model predictive power, design "error", and profitability relative to ideal profits are compared as the amount of market data available increases. We find that even when estimated compensatory models provide relatively good predictive accuracy, they can lead to sub-optimal design decisions when the population uses consideration behavior; convergence of compensatory models to non-compensatory behavior is likely to require unrealistic amounts of data; and modeling heterogeneity in non-compensatory screening is more valuable than heterogeneity in compensatory trade-offs. This supports the claim that designers should carefully identify consideration behaviors before optimizing product portfolios. We also find that higher model predictive power does not necessarily imply better design decisions; that is, different model forms can provide "descriptive" rather than "predictive" information that is useful for design.

The Monod model is a classical microbiological model much used in microbiology, for example to evaluate biodegradation processes. The model describes microbial growth kinetics in batch culture experiments using three parameters: the maximal specific growth rate, the saturation constant and the yield coefficient. However, identification of these parameter values from experimental data is a challenging problem. Recently, it was shown theoretically that the application of optimal design theory in this model is an efficient method for both parameter value identification and economic use of experimental resources (Dette et al., 2003). The purpose of this paper is to provide this method as a computational ?tool? such that it can be used by practitioners-without strong mathematical and statistical backgroundfor the efficient design of experiments in the Monod model. The paper presents careful explanations of the principal theoretical concepts, and a computer program for practical optimal design calculations in Mathematica 5.0 software. In addition, analogous programs in Matlab software will be soon available at www.optimal-design.org. --Monod model,microbial growth,biodegradation kinetics,optimal experimental design,D-optimality

When it is anticipated that data to be collected from an experiment cannot be adequately described by a low-order polynomial, alternative modelling and new design methods are required. Local linear regression, where the response is approximated locally by a series of weighted linear regressions, is an effective nonparametric smoothing method that makes few assumptions about the functional form of the response. We present new methods for the optimal design of experiments for local linear regression, including a new criterion, called DSI-optimality, to find designs that enable precise prediction across a continuous interval. Designs are found numerically for weights defined through the Gaussian and uniform kernels. Theoretical results are presented for the uniform kernel and the special case of prediction at a single point. The sensitivity of the designs to the choice of bandwidth in the local linear regression is studied, and it is found that designs for the Gaussian kernel with large bandwidth have a small number of distinct design points. The methodology is motivated by, and demonstrated on, an experiment from Tribology

We study the problem of optimal subset selection from a set of correlated random variables. In particular, we consider the associated combinatorial optimization problem of maximizing the determinant of a symmetric positive definite matrix that characterizes the chosen subset. This problem arises in many domains, such as experimental designs, regression modeling, and environmental statistics. We establish an efficient polynomial-time algorithm using Determinantal Point Process for approximating the optimal solution to the problem. We demonstrate the advantages of our methods by presenting computational results for both synthetic and real data sets.

We consider the common nonlinear regression model where the variance as well as the mean is a parametric function of the explanatory variables. The c-optimal design problem is investigated in the case when the parameters of both the mean and the variance function are of interest. A geometric characterization of c-optimal designs in this context is presented, which generalizes the classical result of Elfving (1952) for c-optimal designs. As in Elfving's famous characterization c-optimal designs can be described as representations of boundary points of a convex set. However, in the case where there appear parameters of interest in the variance, the structure of the Elfving set is different. Roughly speaking the Elfving set corresponding to a heteroscedastic regression model is the convex hull of a set of ellipsoids induced by the underlying model and indexed by the design space. The c-optimal designs are characterized as representations of the points where the line in direction of the vector c intersects the boundary of the new Elfving set. The theory is illustrated in several examples including pharmacokinetic models with random effects. --c-optimal design,heteroscedastic regression,Elfving's theorem,pharmacokinetic models,random effects,locally optimal design,geometric characterization

We consider a class of convex optimization problems over the simplex of probability measures. Our framework comprises optimal experimental design (OED) problems, in which the measure over the design space indicates which experiments are being selected. Due to the presence of additional bound constraints, the measure possesses a Lebesgue density and the problem can be cast as an optimization problem over the space of essentially bounded functions. For this class of problems, we consider two first-order methods including FISTA and a proximal extrapolated gradient method, along with suitable stopping criteria. Finally, acceleration strategies targeting the dimension of the subproblems in each iteration are discussed. Numerical experiments accompany the analysis throughout the paper.

We present a review of methods for optimal experimental design (OED) for Bayesian inverse problems governed by partial differential equations with infinite-dimensional parameters. The focus is on problems where one seeks to optimize the placement of measurement points, at which data are collected, such that the uncertainty in the estimated parameters is minimized. We present the mathematical foundations of OED in this context and survey the computational methods for the class of OED problems under study. We also outline some directions for future research in this area.

We review the practice of experimental design in the environmental economics literature concerned with choice experiments. We then contrast this with advances in the field of experimental design and present a comparison of statistical efficiency across four different experimental designs evaluated by Monte Carlo experiments. Two different situations are envisaged. First, a correct a priori knowledge of the multinomial logit specification used to derive the design and then an incorrect one. The data generating process is based on estimates from data of a real choice experiment with which preference for rural landscape attributes were studied. Results indicate the D-optimal designs are promising, especially those based on Bayesian algorithms with informative prior. However, if good a priori information is lacking, and if there is strong uncertainty about the real data generating process - conditions which are quite common in environmental valuation - then practitioners might be better off with conventional fractional designs from linear models. Under misspecification, a design of this type produces less biased estimates than its competitors

Optimal experimental design (OED) seeks experiments expected to yield the most useful data for some purpose. In practical circumstances where experiments are time-consuming or resource-intensive, OED can yield enormous savings. We pursue OED for nonlinear systems from a Bayesian perspective, with the goal of choosing experiments that are optimal for parameter inference. Our objective in this context is the expected information gain in model parameters, which in general can only be estimated using Monte Carlo methods. Maximizing this objective thus becomes a stochastic optimization problem. This paper develops gradient-based stochastic optimization methods for the design of experiments on a continuous parameter space. Given a Monte Carlo estimator of expected information gain, we use infinitesimal perturbation analysis to derive gradients of this estimator. We are then able to formulate two gradient-based stochastic optimization approaches: (i) Robbins-Monro stochastic approximation, and (ii) sample average approximation combined with a deterministic quasi-Newton method. A polynomial chaos approximation of the forward model accelerates objective and gradient evaluations in both cases. We discuss the implementation of these optimization methods, then conduct an empirical comparison of their performance. To demonstrate design in a nonlinear setting with partial differential equation forward models, we use the problem of sensor placement for source inversion. Numerical results yield useful guidelines on the choice of algorithm and sample sizes, assess the impact of estimator bias, and quantify tradeoffs of computational cost versus solution quality and robustness.

We present a mathematical framework and computational methods to optimally design a finite number of sequential experiments. We formulate this sequential optimal experimental design (sOED) problem as a finite-horizon partially observable Markov decision process (POMDP) in a Bayesian setting and with information-theoretic utilities. It is built to accommodate continuous random variables, general non-Gaussian posteriors, and expensive nonlinear forward models. sOED then seeks an optimal design policy that incorporates elements of both feedback and lookahead, generalizing the suboptimal batch and greedy designs. We solve for the sOED policy numerically via policy gradient (PG) methods from reinforcement learning, and derive and prove the PG expression for sOED. Adopting an actor-critic approach, we parameterize the policy and value functions using deep neural networks and improve them using gradient estimates produced from simulated episodes of designs and observations. The overall PG-sOED method is validated on a linear-Gaussian benchmark, and its advantages over batch and greedy designs are demonstrated through a contaminant source inversion problem in a convection-diffusion field.

In dose response studies, the dose range is often restricted due to concerns over drug toxicity and/or efficacy. We present restricted and unrestricted interval locally optimal designs with respect to a very general class of optimality criteria for estimating the underlying dose response curve. The underlying curve belongs to a diversified set of link functions suitable for the dose response studies and having a common canonical form. These include the fundamental binary response models – the logit and the probit as well as the skewed versions of these models. The results are illustrated through the re-design of a dose ranging trial conducted at the Merck Research Laboratories (Zeng and Zhu, 1997). This work is a generalization of the results of Dai and Zhu (2002) in terms of the design interval, the underlying dose response curve and the optimality criterion. --Binary response model,Dose ranging,Dose response,Link function,General Equivalence Theorem,Locally compound optimal design

Optimal experimental design is a well studied field in applied science and engineering. Techniques for estimating such a design are commonly used within the framework of parameter estimation. Nonetheless, in recent years parameter estimation techniques are changing rapidly with the introduction of deep learning techniques to replace traditional estimation methods. This in turn requires the adaptation of optimal experimental design that is associated with these new techniques. In this paper we investigate a new experimental design methodology that uses deep learning. We show that the training of a network as a Likelihood Free Estimator can be used to significantly simplify the design process and circumvent the need for the computationally expensive bi-level optimization problem that is inherent in optimal experimental design for non-linear systems. Furthermore, deep design improves the quality of the recovery process for parameter estimation problems. As proof of concept we apply our methodology to two different systems of Ordinary Differential Equations.