\subsection{Design of Experiment Theory (DOE)}

Factorial designs are used to study the effects of two or more factors
simultaneously. One of the most commonly used factorial design family
is called {\it full factorial designs}. Given a set of factors and their
settings, a full factorial design is an experimental
plan which contains all combinations of factor settings. Take a
factorial experiment which involves $k$ factors each at two levels as an
example. The full factorial design for this experiment would have
$2 * 2 * ... * 2 = 2^k$ entries. The experimental plan is then
carried out by making observations at each combination of factor
settings, analyzing the collected data, and computing the factorial
effects.

The fundamental downside of full factorial designs is their run-size
(number of observations required) inefficiency. In the example given
above, even for small values of  $k$, the $2^k$ design can quickly
become very large. In such situations where full factorial designs are
infeasible or simply undesirable, {\it screening designs} are commonly used.
  
Screening designs are economical experimental plans whose primary
purpose is to identify significant main effects ({\em i.e.},
first-order effects) and/or lower-order interaction effects (such as
second and third-order effects). An $n$-order effect is an effect
which is caused by interaction of $n$ factors. Screening designs
reduce run-sizes by aliasing the effects of lower-order interactions
with higher-order ones.

Screening designs leverages two fundamental principles of factorial
experiments: (1) Hierarchical ordering principle and (2) Effect sparsity
principle. These are both empirical
principles whose validities have been confirmed by many real
experiments by the DOE community.

{\bf Hierarchical ordering principle:} 

\begin{itemize}

\item Lower-order effects are more likely to be important than higher
  order effects.

\item Effects of the same order are equally likely to be important.

\end{itemize} 
 
This principle suggests that when resources are scarce, priority
should be given to estimation of lower-order effects.

{\bf Effect sparsity principle:} The number of relatively important
effects in a factorial experiment is small. This principle is also
known as {/it Pareto principle in experimental design}. It suggests
that the experiment designs should  focus on ``vital few'' and not the
``trivial many''.

Screening designs have been extensively used to understand and
improve products and processes developed in manufacturing,
engineering, and physical sciences. In this work, we use them to
understand and improve software quality.


