## Overview

*Designing Experiments and Analyzing Data: A Model Comparison Perspective* (3^{rd }edition) offers an integrative conceptual framework for understanding experimental design and data analysis. The authors (Scott E. Maxwell, Harold D. Delaney, and Ken Kelley) first apply fundamental principles to simple experimental designs followed by an application of the same principles to more complicated designs. Their integrative conceptual framework better prepares readers to understand the logic behind a general strategy of data analysis that is appropriate for a wide variety of designs, which allows for the introduction of more complex topics that are generally omitted from other books. Numerous pedagogical features further facilitate understanding: examples of published research demonstrate the applicability of each chapter’s content; flowcharts assist in choosing the most appropriate procedure; end-of-chapter lists of important formulas highlight key ideas and assist readers in locating the initial presentation of equations; useful programming code and tips are provided throughout the book and in associated resources available on this website (*DesigningExperiments.com*), and extensive sets of exercises help develop a deeper understanding of the subject. Detailed solutions for some of the exercises and realistic data sets are also provided on this website. The pedagogical approach used throughout the book enables readers to gain an overview of experimental design, from conceptualization of the research question to analysis of the data. We aim for the book to be useful for students and researchers seeking the optimal way to design their studies and analyze the resulting data.

The book’s web page at Routledge is here.

Instructors considering the book for adoption can request an examination copy here.

## Citation

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). *Designing Experiments **and Analyzing Data: A Model Comparison Perspective* (3rd ed.). New York: Routledge.

## About the Authors

Scott E. Maxwell is the Fitzsimons Professor of Psychology at the University of Notre Dame. His research interests are in the areas of research methodology and applied behavioral statistics, with much of his recent work focusing on statistical power and accuracy in parameter estimation, especially in randomized designs. He has served as editor of *Psychological Methods*; received the Samuel J. Messick Award for Distinguished Scientific Contributions by the American Psychological Association’s Division of Evaluation, Measurement, and Statistics; and has received multiple teaching awards.

Harold D. Delaney is Emeritus Professor of Psychology at the University of New Mexico, where he received the University’s Outstanding Graduate Teacher of the Year award for his course on experimental design and analysis, and where he directed the Psychology Honors program for 30 years. His research interests in applied statistics include methods that accommodate individual differences among people. He received a Fulbright Award from the U.S. Department of State to spend an academic year lecturing in Budapest, Hungary, and continues to offer courses there.

Ken Kelley is Professor of Information Technology, Analytics, and Operations and the Associate Dean for Faculty and Research in the Mendoza College of Business at the University of Notre Dame. His work is on quantitative methodology, where he focuses on the development, improvement, and evaluation of statistical methods and measurement issues. He is an Accredited Professional Statistician (PStat^{®}); recipient of the Anne Anastasi early career award by the Evaluation, Measurement, & Statistics Division of the APA, and previously served as an associate editor of *Psychological Methods*.

## Table of Contents

*Part 1: Conceptual Basis of Experimental Design*

**Chapter 1: The Logic of Experimental Design and Analysis
**Overview of Chapter: Research Questions Addressed

Published Example

Philosophy of Science

– The Traditional View of Science

– Responses to the Criticisms of the Idea of Pure Science

— Assumptions

— Modern Philosophy of Science

Introduction to the Fisher Tradition

– “Interpretation and Its Reasoned Basis”

– A Discrete Probability Example

– Randomization Test

**– Of Hypotheses and**

*p*Values: Fisher Versus Neyman-Pearson

**– Toward Tests Based on Distributional Assumptions**

**— Statistical Tests With Convenience Samples**

**— The Assumption of Normality**

**Summary of Main Points**

**Important Formulas**

**Online Materials Available on**

*DesigningExperiments.com*

**Exercises**

**Chapter 2: Drawing Valid Inferences From Experiments
**Overview of Chapter: Research Questions Addressed

Published Example

Threats to the Validity of Inferences From Experiments

– Types of Validity

— Statistical Conclusion Validity

— Internal Validity

— Construct Validity

— External Validity

-Conceptualizing and Controlling for Threats to Validity

Overview of Experimental Designs to Be Considered

Summary of Main Points

**Exercises**

**Part II Model Comparisons for Between Subjects Designs**

**Chapter 3: Introduction to Model Comparisons: One-Way Between-Subjects Designs
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

The General Linear Model

One-Group Situation

– Basics of Models

– Optional

— Proof That

*Y*Is the Least-Squares Estimate of

*μ*

– Development of the General Form of the Test Statistic

– Numerical Example

– Relationship of Models and Hypotheses

Two-Group Situation

– Development in Terms of Models

– Alternative Development and Identification With Traditional Terminology

The General Case of One-Way Designs

– Formulation in Terms of Models

– Numerical Example

– A Model in Terms of Effects

— Parameter Estimates

— Computation of the Test Statistic

On Tests of Significance and Measures of Effect

Measures of Effect

– Measures of Effect Size

— Mean Difference

— Confidence Intervals

— Estimated Effect Parameters

— The Standardized Difference Between Means

— Confidence Intervals for Standardized Differences Between Means

— Standardized Effects, and the Signal-to-Noise Ratio

– Measures of Association Strength

— Confidence Intervals for Measures of Association Strength

— Evaluation of Measures

– Alternative Representations of Effects

— Binomial Effect Size Display (BESD)

— Common Language (CL) Effect Size

— Graphical Methods

– Statistical Assumptions

— Implications for Expected Values

— Robustness of ANOVA

— Checking for Normality and Homogeneity of Variance

— Transformations

Power of the

*F*Test: One-Way ANOVA

– Determining an Appropriate Sample Size

— Specifying the Minimally Important Difference

— Specifying Population Parameters and Using Power Charts

— Determining Sample Size Using

*δ*and Table 3.10

— Pilot Data and Observed Power

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 4: Individual Comparisons of Means
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

A Model Comparison Approach for Testing Individual Comparisons

– Preview of Individual Comparisons

– Relationship to Model Comparisons

– Expression of

*F*Statistic

– Numerical Example

Complex Comparisons

– Models Perspective

– Numerical Example

The

*t*Test Formulation of Hypothesis Testing for Contrasts

– Practical Implications

Unequal Population Variances

– Numerical Example

– Practical Implications

Measures of Effect

– Measures of Effect Size

— Confidence Intervals

— Standardized Difference

– Measures of Association Strength

Testing More Than One Contrast

– How Many Contrasts Should Be Tested?

– Linear Independence of Contrasts

– Orthogonality of Contrasts

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 5: Testing Several Contrasts: The Multiple-Comparisons Problem
**Overview of Chapter: Research Questions Addressed

Published Example

Multiple Comparisons

– Experimentwise and Per-Comparison Error Rates

Simultaneous Confidence Intervals

– Levels of Strength of Inference

– Types of Contrasts

– Overview of Techniques

– Planned Versus Post Hoc Contrasts

Multiple Planned Comparisons

– Bonferroni Adjustment

– Modification of the Bonferroni Approach With Unequal Variances

– Numerical Example

Pairwise Comparisons

– Tukey’s HSD Procedure

– Modifications of Tukey’s HSD

– Numerical Example

Post Hoc Complex Comparisons

– Proof That

*SS_*max = SS_B

– Comparison of Scheffé to Bonferroni and Tukey

– Modifications of Scheffé’s Method

– Numerical Example

Other Multiple-Comparison Procedures

– Dunnett’s Procedure for Comparisons With a Control

– Numerical Example

– Procedures for Comparisons With the Best

– Numerical Example

– Fisher’s LSD (Protected

*t*)

False Discovery Rate

Choosing an Appropriate Procedure

Summary of Main Points

Important Formulas

Online Materials Available at

*DesigningExperiments.com*

Exercises

**Chapter 6: Trend Analysis
**Overview of Chapter: Research Questions Addressed

Published Example

Quantitative Factors

Statistical Treatment of Trend Analysis

– The Slope Parameter

– Numerical Example

– Hypothesis Test of Slope Parameter

Confidence Interval and Other Effect Size Measures for the Slope Parameter

– Numerical Example

Testing for Nonlinearity

– Numerical Example

Testing Individual Higher Order Trends

– Contrast Coefficients for Higher Order Trends

– Numerical Example

Further Examination of Nonlinear Trends

Trend Analysis With Unequal Sample Sizes

Concluding Comments

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 7: Two-Way Between-Subjects Factorial Designs
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

The 2 × 2 Design

– The Concept of Interaction

– Additional Perspectives on the Interaction

A Model Comparison Approach to the General Two-Factor Design

– Alternate Form of Full Model

– Comparison of Models for Hypothesis Testing

– Numerical Example

– Familywise Control of Alpha Level

– Measures of Effect

Follow-Up Tests

– Further Investigation of Main Effects

– Further Investigation of an Interaction—Simple Effects

— Relationships of Main Effect, Interaction, and Simple Effects

— Consideration of Type I Error Rate in Testing Simple Effects

— Error Term for Testing Simple Effects

– An Alternative Method for Investigating an Interaction—Interaction Contrasts

Statistical Power

Advantages of Factorial Designs

Nonorthogonal Designs

– Design Considerations

– Relationship Between Design and Analysis

– Analysis of the 2 × 2 Nonorthogonal Design

– Test of the Interaction

– Unweighted Marginal Means and Type III Sum of Squares

– Unweighted Versus Weighted Marginal Means

– Type II Sum of Squares

– Summary of Three Types of Sum of Squares

Analysis of the General

*a*×

*b*Nonorthogonal Design

– Test of the Interaction

– Test of Unweighted Marginal Means

– Test of Marginal Means in an Additive Model

– Test of Weighted Marginal Means

– Summary of Types of Sum of Squares

– Which Type of Sum of Squares Is Best?

– A Note on Statistical Software for Analyzing Nonorthogonal Designs

– Numerical Example

– Final Remarks

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 8: Higher-Order Between-Subjects Factorial Designs
**Overview of Chapter: Research Questions Addressed

Published Example

The 2 × 2 × 2 Design

– The Meaning of Main Effects

– The Meaning of Two-Way Interactions

– The Meaning of the Three-Way Interaction

– Graphical Depiction

– Further Consideration of the Three-Way Interaction

– Summary of Meaning of Effects

The General

*A*×

*B*×

*C*Design

– The Full Model

– Formulation of Restricted Models

**Numerical Example**

**– Implications of a Three-Way Interaction**

– General Guideline for Analyzing Effects

– Summary of Results

– Graphical Depiction of Data

– Confidence Intervals for Single Degree of Freedom Effects

– Other Questions of Potential Interest

– Tests to Be Performed When the Three-Way Interaction Is Non-Significant

– Nonorthogonal Designs

Higher Order Designs

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 9: Designs With Covariates: ANCOVA and Blocking
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

ANCOVA

– The Logic of ANCOVA

– Linear Models for ANCOVA

— Parameter Estimates

— Comparison of Models

– Two Consequences of Using ANCOVA

— Test of Regression

— Estimated Conditional Means

— Examples of Adjusted Effects

— Summary

– Assumptions in ANCOVA

— Basic Implications

— Lack of Independence of Treatment and Covariate

— Summary Regarding Lack of Independence of Treatment and Covariate

**— Measurement Error in Covariate**

**– Numerical Example**

**– Measures of Effect**

**– Comparisons Among Adjusted Group Means**

**– Generalizations of the ANCOVA Model**

**— Multiple Covariates**

**— Nonlinear Relationships**

**— Multifactor Studies**

**– Choosing Covariates in Randomized Designs**

**– Sample Size Planning and Power Analysis in ANCOVA**

**Alternate Methods of Analyzing Designs With Concomitant Variables**

**– ANOVA of Residuals**

**– Gain Scores**

**– Blocking**

**— Conclusions Regarding Blocking**

**– Matching: Propensity Scores**

**Summary of Main Points**

**Important Formulas**

**Online Materials Available on**

*DesigningExperiments.com*

**Exercises**

**Extension: Heterogeneity of Regression**

**Test for Heterogeneity of Regression**

**Accommodating Heterogeneity of Regression**

**– Simultaneous Tests**

**– Carrying Out Tests and Determining Regions of Significance**

**— Summary Regarding Heterogeneity of Regression**

**Important Formulas**

**Exercises**

**Chapter 10: Designs With Random or Nested Factors
**Overview of Chapter: Research Questions Addressed

Designs With Random Factors

**– Introduction to Random Effects**

**– One-Factor Case**

**— Model**

**— Model Comparisons**

**— Expected Values**

**– Two-Factor Case**

**— Expected Mean Squares**

**— Model Comparisons**

**— Selection of Error Terms**

**– Numerical Example**

**– Alternative Tests and Design Considerations With Random Factors**

**– Follow-Up Tests and Confidence Intervals**

**– Measures of Association Strength**

**— Intraclass Correlation**

**— Numerical Example**

**– Using Statistical Computer Programs to Analyze Designs With Random Factors**

**– Determining Power in Designs With Random Factors**

**Designs With Nested Factors**

**– Introduction to Nested Factors**

**– Example**

**– Models and Tests**

**– Degrees of Freedom**

**– Statistical Assumptions and Related Issues**

**– Follow-Up Tests and Confidence Intervals**

**– Standardized Effect Size Estimates**

**– Strength of Association in Nested Designs**

**– Using Statistical Computer Programs to Analyze Nested Designs**

**– Selection of Error Terms When Nested Factors Are Present**

**– Complications That Arise in More Complex Designs**

**Summary of Main Points**

**Important Formulas**

**Online Materials Available on**

*DesigningExperiments.com*

**Exercises**

*Part III: Model Comparisons for Designs Involving Within-Subjects Factors*

**Chapter 11: ****One-Way Within-Subjects Designs: Univariate Approach
**Overview of Chapter: Research Questions Addressed

**Published Example**

**Prototypical Within-Subjects Designs**

**Advantages of Within-Subjects Designs**

**Analysis of Repeated-Measures Designs With Two Levels**

– The Problem of Correlated Errors

**– Reformulation of Model**

**Analysis of Within-Subjects Designs With More Than Two Levels**

**Traditional Univariate (Mixed-Model) Approach**

**– Comparison of Full and Restricted Models**

**– Estimation of Parameters: Numerical Example**

**Assumptions in the Traditional Univariate (Mixed-Model) Approach**

**– Homogeneity, Sphericity, and Compound Symmetry**

**– Numerical Example**

**Adjusted Univariate Tests**

**– Lower-Bound Adjustment**

*– ε*hat

*Adjustment*

*– ε*tilde

*Adjustment*

**– Summary of Four Mixed-Model Approaches**

**Measures of Effect**

**Comparisons Among Individual Means**

**Confidence Intervals for Comparisons**

**Optional**

**– Confidence Intervals With Pooled and Separate Variances**

**Considerations in Designing Within-Subjects Experiments**

**– Order Effects**

**– Differential Carryover Effects**

**– Controlling for Order Effects With More Than Two Levels: Latin Square Designs**

**Relative Advantages of Between-Subjects and Within-Subjects Designs**

**Intraclass Correlations for Assessing Reliability**

**Summary of Main Points**

**Important Formulas**

**Online Materials Available on**

*DesigningExperiments.com*

**Exercises**

**Chapter 12: Higher-Order Designs With Within-Subjects Factors: Univariate Approach
**Overview of Chapter: Research Questions Addressed

Published Example

Designs With Two Within-Subjects Factors

– Omnibus Tests

– Numerical Example

– Further Investigation of Main Effects

– Further Investigation of an Interaction — Simple Effects

– Interaction Contrasts

– Statistical Packages and Pooled Error Terms Versus Separate Error Terms

– Assumptions

– Adjusted Univariate Tests

– Confidence Intervals

– Quasi-

*F*Ratios

One Within-Subjects Factor and One Between-Subjects Factor in the Same Design

– Omnibus Tests

— An Appropriate Full Model

— Restricted Models

— Error Terms

— Numerical Example

– Further Investigation of Main Effects

— Between-Subjects Factor

— Within-Subjects Factor

– Further Investigation of an Interaction — Simple Effects

— Within-Subjects Effects at a Fixed Level of Between-Subjects Factor

— Between-Subjects Effects at a Fixed Level of Within-Subjects Factor

– Interaction Contrasts

– Assumptions

– Adjusted Univariate Tests

More Complex Designs

– Designs With Additional Factors

– Latin Square Designs

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 13: One-Way Within-Subjects Designs: Multivariate Approach
**Overview of Chapter: Research Questions Addressed

Published Example

A Brief Review of Analysis for Designs With Two Levels

Multivariate Analysis of Within-Subjects Designs With Three Levels

– Need for Multiple

*D*Variables

– Full and Restricted Models

– The Relationship Between

*D*1 and

*D*2

– Matrix Formulation and Determinants

– Test Statistic

Multivariate Analysis of Within-Subjects Designs With

*a*Levels

– Forming

*D*Variables

– Test Statistic

– Numerical Example

Measures of Effect

Choosing an Appropriate Sample Size

Choice of

*D*Variables

Tests of Individual Contrasts

Multiple-Comparison Procedures: Determination of Critical Values

– Planned Comparisons

– Pairwise Comparisons

– Post Hoc Complex Comparisons

Confidence Intervals for Contrasts

The Relationship Between the Multivariate Approach and the Mixed-Model Approach

– Orthonormal Contrasts

– Comparison of the Two Approaches

Multivariate and Mixed-Model Approaches for Testing Contrasts

– Numerical Example

– The Difference in Error Terms

– Which Error Term Is Better?

A General Comparison of the Multivariate and Mixed-Model Approaches

– Assumptions

– Tests of Contrasts

– Type I Error Rates

– Type II Error Rates

– Summary

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Chapter 14: Higher-Order Designs With Within-Subjects Factors: Multivariate Approach
**Overview of Chapter: Research Questions Addressed

– Published Example

Two Within-Subjects Factors, Each With Two Levels

– Formation of Main Effect

*D*Variables

**– Formation of Interaction**

*D*Variables

**– Relationship to the Mixed-Model Approach**

**Multivariate Analysis of Two-Way**

*a*×

*b*Within-Subjects Designs

**– Formation of Main Effect**

*D*Variables

**– Formation of Interaction**

*D*Variables

**– Omnibus Tests — Multivariate Significance Tests**

**Measures of Effect**

– Further Investigation of Main Effects

**– Further Investigation of an Interaction — Simple Effects**

– Interaction Contrasts

Confidence Intervals for Contrasts

– Multivariate and Mixed-Model Approaches for Testing Contrasts

– Comparison of the Multivariate and Mixed-Model Approaches

One Within-Subjects Factor and One Between-Subjects Factor in the Same Design

– Split-Plot Design With Two Levels of the Within-Subjects Factor

— Main Effect of Between-Subjects Factor

— Within-Subjects Effects

— Test of the Interaction

— Within-Subjects Main Effect

— Summary

– General

*a*×

*b*Split-Plot Design

— Between-Subjects Main Effect

— Within-Subjects Effects

— Within-Subjects Main Effect

— Test of the Interaction

Measures of Effect

– Further Investigation of Main Effects

– Further Investigation of an Interaction — Simple Effects

– Between-Subjects Effects at a Fixed Level of Within-Subjects Factor

– Within-Subjects Effects at a Fixed Level of Between-Subjects Factor

– Cell Mean Comparisons

– Interaction Contrasts

Confidence Intervals for Contrasts

Assumptions of the Multivariate Approach

Multivariate and Mixed-Model Approaches for Testing Within-Subjects Contrasts

– Comparison of the Multivariate and Mixed-Model Approaches

Optional

– More Complex Designs

Summary of Main Points

Important Formulas

– Two-Way Within-Subjects Designs

– Split-Plot Designs

Online Materials Available on

*DesigningExperiments.com*

Exercises

*Part IV: Mixed Effects Models*

**Chapter 15: An Introduction to Mixed-Effects Models: Within-Subjects Designs
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

Advantages of Mixed-Effects Models

– Within-Subjects Designs

– Overview of Remainder of Chapter

Within-Subjects Designs

– Various Types of Within-Subjects Designs

– Models for Longitudinal Data

– Review of the ANOVA Mixed-Model Approach

Mixed-Effects Models

– A Maximum Likelihood Approach

– An Example of Maximum Likelihood Estimation

– Comparison of ANOVA and Maximum Likelihood Models

– Numerical Example

– A Closer Look at the Random Effects Model

– Graphical Representation of Longitudinal Data

– Graphical Representation of the Random Intercept Model

– Coding Random Effects Predictor Variables

– Random Effects Parameters

– Numerical Example

– Graphical Representation of a Model With Random Slope and Intercept

– Further Consideration of Competing Models

– Additional Models

– Straight-Line Change Model

– Graphical Representation of a Growth Curve Model

– Design Considerations

An Alternative Approach and Conceptualization

– Additional Covariance Matrix Structures

– Tests of Contrasts

– Overview of Broader Model Comparison

Complex Designs

– Factorial Fixed Effects

– Multiple Variables Measured Over Time

Unbalanced Designs

Summary of Main Points

Important Formulas

**Online Materials Available on**

*DesigningExperiments.com*

Exercises

**Chapter 16: An Introduction to Mixed-Effects Models: Nested Designs
**Overview of Chapter: Research Questions Addressed

Published Example

Introduction

Review of the ANOVA Approach

Mixed-Effects Models Analysis for the Simple Nested Design

– Numerical Example — Equal

*n*

– Numerical Example — Unequal

*n*

Mixed-Effects Models for Complex Nested Designs

– Hierarchical Representation of the Model for a Simple Nested Design

– Models With Additional Level 2 Variables

– Models With Additional Level 1 Variables

Summary of Main Points

Important Formulas

Online Materials Available on

*DesigningExperiments.com*

Exercises

**Appendix (Statistical Tables)**

Table 1: Critical Values of *t* Distribution

Table 2: Critical Values of *F* Distribution

Table 3: Critical Values of Bonferroni* F *Distribution With 1 Numerator *df* and Familywise α of .05

Table 4: Critical Values of Studentized Range Distribution

Table 5: Critical Values of Studentized Maximum Modulus Distribution

Table 6: Critical Values of Dunnett’s Two-Tailed Test for Comparing Treatments to a Control

Table 7: Critical Values of Dunnett’s One-Tailed Test for Comparing Treatments to a Control

Table 8: Critical Values of Bryant-Paulson Generalized Studentized Range

Table 9: Critical Values of Chi-Square Distribution

Table 10: Coefficients of Orthogonal Polynomials

Table 11: Pearson-Hartley Power Charts

**References **

**Name Index **

**Subject Index**

## ISBN Information

ISBN: 978-1-138-89228-6 (hbk)

ISBN: 978-1-315-64295-6 (ebk)