Overview

Designing Experiments and Analyzing Data: A Model Comparison Perspective (3^rd edition) offers an integrative conceptual framework for understanding experimental design and data analysis. The authors (Scott E. Maxwell, Harold D. Delaney, and Ken Kelley) first apply fundamental principles to simple experimental designs followed by an application of the same principles to more complicated designs. Their integrative conceptual framework better prepares readers to understand the logic behind a general strategy of data analysis that is appropriate for a wide variety of designs, which allows for the introduction of more complex topics that are generally omitted from other books. Numerous pedagogical features further facilitate understanding: examples of published research demonstrate the applicability of each chapter’s content; flowcharts assist in choosing the most appropriate procedure; end-of-chapter lists of important formulas highlight key ideas and assist readers in locating the initial presentation of equations; useful programming code and tips are provided throughout the book and in associated resources available on this website (DesigningExperiments.com), and extensive sets of exercises help develop a deeper understanding of the subject. Detailed solutions for some of the exercises and realistic data sets are also provided on this website. The pedagogical approach used throughout the book enables readers to gain an overview of experimental design, from conceptualization of the research question to analysis of the data. We aim for the book to be useful for students and researchers seeking the optimal way to design their studies and analyze the resulting data.

The book’s web page at Routledge is here.
Instructors considering the book for adoption can request an examination copy here.

Citation

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). Designing Experiments and Analyzing Data: A Model Comparison Perspective (3rd ed.). New York: Routledge.

About the Authors

Scott E. Maxwell is the Fitzsimons Professor of Psychology at the University of Notre Dame. His research interests are in the areas of research methodology and applied behavioral statistics, with much of his recent work focusing on statistical power and accuracy in parameter estimation, especially in randomized designs. He has served as editor of Psychological Methods; received the Samuel J. Messick Award for Distinguished Scientific Contributions by the American Psychological Association’s Division of Evaluation, Measurement, and Statistics; and has received multiple teaching awards.

Harold D. Delaney is Emeritus Professor of Psychology at the University of New Mexico, where he received the University’s Outstanding Graduate Teacher of the Year award for his course on experimental design and analysis, and where he directed the Psychology Honors program for 30 years. His research interests in applied statistics include methods that accommodate individual differences among people. He received a Fulbright Award from the U.S. Department of State to spend an academic year lecturing in Budapest, Hungary, and continues to offer courses there.

Ken Kelley is Professor of Information Technology, Analytics, and Operations and the Associate Dean for Faculty and Research in the Mendoza College of Business at the University of Notre Dame. His work is on quantitative methodology, where he focuses on the development, improvement, and evaluation of statistical methods and measurement issues. He is an Accredited Professional Statistician (PStat^®); recipient of the Anne Anastasi early career award by the Evaluation, Measurement, & Statistics Division of the APA, and previously served as an associate editor of Psychological Methods.

Table of Contents

Part 1: Conceptual Basis of Experimental Design
Chapter 1: The Logic of Experimental Design and Analysis
Overview of Chapter: Research Questions Addressed
Published Example
Philosophy of Science
– The Traditional View of Science
– Responses to the Criticisms of the Idea of Pure Science
— Assumptions
— Modern Philosophy of Science
Introduction to the Fisher Tradition
– “Interpretation and Its Reasoned Basis”
– A Discrete Probability Example
– Randomization Test
– Of Hypotheses and p Values: Fisher Versus Neyman-Pearson
– Toward Tests Based on Distributional Assumptions
— Statistical Tests With Convenience Samples
— The Assumption of Normality
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 2: Drawing Valid Inferences From Experiments
Overview of Chapter: Research Questions Addressed
Published Example
Threats to the Validity of Inferences From Experiments
– Types of Validity
— Statistical Conclusion Validity
— Internal Validity
— Construct Validity
— External Validity
-Conceptualizing and Controlling for Threats to Validity
Overview of Experimental Designs to Be Considered
Summary of Main Points
Exercises

Part II Model Comparisons for Between Subjects Designs
Chapter 3: Introduction to Model Comparisons: One-Way Between-Subjects Designs
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
The General Linear Model
One-Group Situation
– Basics of Models
– Optional
— Proof That Y Is the Least-Squares Estimate of μ
– Development of the General Form of the Test Statistic
– Numerical Example
– Relationship of Models and Hypotheses
Two-Group Situation
– Development in Terms of Models
– Alternative Development and Identification With Traditional Terminology
The General Case of One-Way Designs
– Formulation in Terms of Models
– Numerical Example
– A Model in Terms of Effects
— Parameter Estimates
— Computation of the Test Statistic
On Tests of Significance and Measures of Effect
Measures of Effect
– Measures of Effect Size
— Mean Difference
— Confidence Intervals
— Estimated Effect Parameters
— The Standardized Difference Between Means
— Confidence Intervals for Standardized Differences Between Means
— Standardized Effects, and the Signal-to-Noise Ratio
– Measures of Association Strength
— Confidence Intervals for Measures of Association Strength
— Evaluation of Measures
– Alternative Representations of Effects
— Binomial Effect Size Display (BESD)
— Common Language (CL) Effect Size
— Graphical Methods
– Statistical Assumptions
— Implications for Expected Values
— Robustness of ANOVA
— Checking for Normality and Homogeneity of Variance
— Transformations
Power of the F Test: One-Way ANOVA
– Determining an Appropriate Sample Size
— Specifying the Minimally Important Difference
— Specifying Population Parameters and Using Power Charts
— Determining Sample Size Using δ and Table 3.10
— Pilot Data and Observed Power
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 4: Individual Comparisons of Means
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
A Model Comparison Approach for Testing Individual Comparisons
– Preview of Individual Comparisons
– Relationship to Model Comparisons
– Expression of F Statistic
– Numerical Example
Complex Comparisons
– Models Perspective
– Numerical Example
The t Test Formulation of Hypothesis Testing for Contrasts
– Practical Implications
Unequal Population Variances
– Numerical Example
– Practical Implications
Measures of Effect
– Measures of Effect Size
— Confidence Intervals
— Standardized Difference
– Measures of Association Strength
Testing More Than One Contrast
– How Many Contrasts Should Be Tested?
– Linear Independence of Contrasts
– Orthogonality of Contrasts
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 5: Testing Several Contrasts: The Multiple-Comparisons Problem
Overview of Chapter: Research Questions Addressed
Published Example
Multiple Comparisons
– Experimentwise and Per-Comparison Error Rates
Simultaneous Confidence Intervals
– Levels of Strength of Inference
– Types of Contrasts
– Overview of Techniques
– Planned Versus Post Hoc Contrasts
Multiple Planned Comparisons
– Bonferroni Adjustment
– Modification of the Bonferroni Approach With Unequal Variances
– Numerical Example
Pairwise Comparisons
– Tukey’s HSD Procedure
– Modifications of Tukey’s HSD
– Numerical Example
Post Hoc Complex Comparisons
– Proof That SS_max = SS_B
– Comparison of Scheffé to Bonferroni and Tukey
– Modifications of Scheffé’s Method
– Numerical Example
Other Multiple-Comparison Procedures
– Dunnett’s Procedure for Comparisons With a Control
– Numerical Example
– Procedures for Comparisons With the Best
– Numerical Example
– Fisher’s LSD (Protected t)
False Discovery Rate
Choosing an Appropriate Procedure
Summary of Main Points
Important Formulas
Online Materials Available at DesigningExperiments.com
Exercises

Chapter 6: Trend Analysis
Overview of Chapter: Research Questions Addressed
Published Example
Quantitative Factors
Statistical Treatment of Trend Analysis
– The Slope Parameter
– Numerical Example
– Hypothesis Test of Slope Parameter
Confidence Interval and Other Effect Size Measures for the Slope Parameter
– Numerical Example
Testing for Nonlinearity
– Numerical Example
Testing Individual Higher Order Trends
– Contrast Coefficients for Higher Order Trends
– Numerical Example
Further Examination of Nonlinear Trends
Trend Analysis With Unequal Sample Sizes
Concluding Comments
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 7: Two-Way Between-Subjects Factorial Designs
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
The 2 × 2 Design
– The Concept of Interaction
– Additional Perspectives on the Interaction
A Model Comparison Approach to the General Two-Factor Design
– Alternate Form of Full Model
– Comparison of Models for Hypothesis Testing
– Numerical Example
– Familywise Control of Alpha Level
– Measures of Effect
Follow-Up Tests
– Further Investigation of Main Effects
– Further Investigation of an Interaction—Simple Effects
— Relationships of Main Effect, Interaction, and Simple Effects
— Consideration of Type I Error Rate in Testing Simple Effects
— Error Term for Testing Simple Effects
– An Alternative Method for Investigating an Interaction—Interaction Contrasts
Statistical Power
Advantages of Factorial Designs
Nonorthogonal Designs
– Design Considerations
– Relationship Between Design and Analysis
– Analysis of the 2 × 2 Nonorthogonal Design
– Test of the Interaction
– Unweighted Marginal Means and Type III Sum of Squares
– Unweighted Versus Weighted Marginal Means
– Type II Sum of Squares
– Summary of Three Types of Sum of Squares
Analysis of the General a × b Nonorthogonal Design
– Test of the Interaction
– Test of Unweighted Marginal Means
– Test of Marginal Means in an Additive Model
– Test of Weighted Marginal Means
– Summary of Types of Sum of Squares
– Which Type of Sum of Squares Is Best?
– A Note on Statistical Software for Analyzing Nonorthogonal Designs
– Numerical Example
– Final Remarks
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 8: Higher-Order Between-Subjects Factorial Designs
Overview of Chapter: Research Questions Addressed
Published Example
The 2 × 2 × 2 Design
– The Meaning of Main Effects
– The Meaning of Two-Way Interactions
– The Meaning of the Three-Way Interaction
– Graphical Depiction
– Further Consideration of the Three-Way Interaction
– Summary of Meaning of Effects
The General A × B × C Design
– The Full Model
– Formulation of Restricted Models
Numerical Example
– Implications of a Three-Way Interaction
– General Guideline for Analyzing Effects
– Summary of Results
– Graphical Depiction of Data
– Confidence Intervals for Single Degree of Freedom Effects
– Other Questions of Potential Interest
– Tests to Be Performed When the Three-Way Interaction Is Non-Significant
– Nonorthogonal Designs
Higher Order Designs
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 9: Designs With Covariates: ANCOVA and Blocking
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
ANCOVA
– The Logic of ANCOVA
– Linear Models for ANCOVA
— Parameter Estimates
— Comparison of Models
– Two Consequences of Using ANCOVA
— Test of Regression
— Estimated Conditional Means
— Examples of Adjusted Effects
— Summary
– Assumptions in ANCOVA
— Basic Implications
— Lack of Independence of Treatment and Covariate
— Summary Regarding Lack of Independence of Treatment and Covariate
— Measurement Error in Covariate
– Numerical Example
– Measures of Effect
– Comparisons Among Adjusted Group Means
– Generalizations of the ANCOVA Model
— Multiple Covariates
— Nonlinear Relationships
— Multifactor Studies
– Choosing Covariates in Randomized Designs
– Sample Size Planning and Power Analysis in ANCOVA
Alternate Methods of Analyzing Designs With Concomitant Variables
– ANOVA of Residuals
– Gain Scores
– Blocking
— Conclusions Regarding Blocking
– Matching: Propensity Scores
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises
Extension: Heterogeneity of Regression
Test for Heterogeneity of Regression
Accommodating Heterogeneity of Regression
– Simultaneous Tests
– Carrying Out Tests and Determining Regions of Significance
— Summary Regarding Heterogeneity of Regression
Important Formulas
Exercises

Chapter 10: Designs With Random or Nested Factors
Overview of Chapter: Research Questions Addressed
Designs With Random Factors
– Introduction to Random Effects
– One-Factor Case
— Model
— Model Comparisons
— Expected Values
– Two-Factor Case
— Expected Mean Squares
— Model Comparisons
— Selection of Error Terms
– Numerical Example
– Alternative Tests and Design Considerations With Random Factors
– Follow-Up Tests and Confidence Intervals
– Measures of Association Strength
— Intraclass Correlation
— Numerical Example
– Using Statistical Computer Programs to Analyze Designs With Random Factors
– Determining Power in Designs With Random Factors
Designs With Nested Factors
– Introduction to Nested Factors
– Example
– Models and Tests
– Degrees of Freedom
– Statistical Assumptions and Related Issues
– Follow-Up Tests and Confidence Intervals
– Standardized Effect Size Estimates
– Strength of Association in Nested Designs
– Using Statistical Computer Programs to Analyze Nested Designs
– Selection of Error Terms When Nested Factors Are Present
– Complications That Arise in More Complex Designs
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Part III: Model Comparisons for Designs Involving Within-Subjects Factors

Chapter 11: One-Way Within-Subjects Designs: Univariate Approach
Overview of Chapter: Research Questions Addressed
Published Example
Prototypical Within-Subjects Designs
Advantages of Within-Subjects Designs
Analysis of Repeated-Measures Designs With Two Levels
– The Problem of Correlated Errors
– Reformulation of Model
Analysis of Within-Subjects Designs With More Than Two Levels
Traditional Univariate (Mixed-Model) Approach
– Comparison of Full and Restricted Models
– Estimation of Parameters: Numerical Example
Assumptions in the Traditional Univariate (Mixed-Model) Approach
– Homogeneity, Sphericity, and Compound Symmetry
– Numerical Example
Adjusted Univariate Tests
– Lower-Bound Adjustment
– ε hat Adjustment
– ε tilde Adjustment
– Summary of Four Mixed-Model Approaches
Measures of Effect
Comparisons Among Individual Means
Confidence Intervals for Comparisons
Optional
– Confidence Intervals With Pooled and Separate Variances
Considerations in Designing Within-Subjects Experiments
– Order Effects
– Differential Carryover Effects
– Controlling for Order Effects With More Than Two Levels: Latin Square Designs
Relative Advantages of Between-Subjects and Within-Subjects Designs
Intraclass Correlations for Assessing Reliability
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 12: Higher-Order Designs With Within-Subjects Factors: Univariate Approach
Overview of Chapter: Research Questions Addressed
Published Example
Designs With Two Within-Subjects Factors
– Omnibus Tests
– Numerical Example
– Further Investigation of Main Effects
– Further Investigation of an Interaction — Simple Effects
– Interaction Contrasts
– Statistical Packages and Pooled Error Terms Versus Separate Error Terms
– Assumptions
– Adjusted Univariate Tests
– Confidence Intervals
– Quasi-F Ratios
One Within-Subjects Factor and One Between-Subjects Factor in the Same Design
– Omnibus Tests
— An Appropriate Full Model
— Restricted Models
— Error Terms
— Numerical Example
– Further Investigation of Main Effects
— Between-Subjects Factor
— Within-Subjects Factor
– Further Investigation of an Interaction — Simple Effects
— Within-Subjects Effects at a Fixed Level of Between-Subjects Factor
— Between-Subjects Effects at a Fixed Level of Within-Subjects Factor
– Interaction Contrasts
– Assumptions
– Adjusted Univariate Tests
More Complex Designs
– Designs With Additional Factors
– Latin Square Designs
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 13: One-Way Within-Subjects Designs: Multivariate Approach
Overview of Chapter: Research Questions Addressed
Published Example
A Brief Review of Analysis for Designs With Two Levels
Multivariate Analysis of Within-Subjects Designs With Three Levels
– Need for Multiple D Variables
– Full and Restricted Models
– The Relationship Between D1 and D2
– Matrix Formulation and Determinants
– Test Statistic
Multivariate Analysis of Within-Subjects Designs With a Levels
– Forming D Variables
– Test Statistic
– Numerical Example
Measures of Effect
Choosing an Appropriate Sample Size
Choice of D Variables
Tests of Individual Contrasts
Multiple-Comparison Procedures: Determination of Critical Values
– Planned Comparisons
– Pairwise Comparisons
– Post Hoc Complex Comparisons
Confidence Intervals for Contrasts
The Relationship Between the Multivariate Approach and the Mixed-Model Approach
– Orthonormal Contrasts
– Comparison of the Two Approaches
Multivariate and Mixed-Model Approaches for Testing Contrasts
– Numerical Example
– The Difference in Error Terms
– Which Error Term Is Better?
A General Comparison of the Multivariate and Mixed-Model Approaches
– Assumptions
– Tests of Contrasts
– Type I Error Rates
– Type II Error Rates
– Summary
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 14: Higher-Order Designs With Within-Subjects Factors: Multivariate Approach
Overview of Chapter: Research Questions Addressed
– Published Example
Two Within-Subjects Factors, Each With Two Levels
– Formation of Main Effect D Variables
– Formation of Interaction D Variables
– Relationship to the Mixed-Model Approach
Multivariate Analysis of Two-Way a × b Within-Subjects Designs
– Formation of Main Effect D Variables
– Formation of Interaction D Variables
– Omnibus Tests — Multivariate Significance Tests
Measures of Effect
– Further Investigation of Main Effects
– Further Investigation of an Interaction — Simple Effects
– Interaction Contrasts
Confidence Intervals for Contrasts
– Multivariate and Mixed-Model Approaches for Testing Contrasts
– Comparison of the Multivariate and Mixed-Model Approaches
One Within-Subjects Factor and One Between-Subjects Factor in the Same Design
– Split-Plot Design With Two Levels of the Within-Subjects Factor
— Main Effect of Between-Subjects Factor
— Within-Subjects Effects
— Test of the Interaction
— Within-Subjects Main Effect
— Summary
– General a × b Split-Plot Design
— Between-Subjects Main Effect
— Within-Subjects Effects
— Within-Subjects Main Effect
— Test of the Interaction
Measures of Effect
– Further Investigation of Main Effects
– Further Investigation of an Interaction — Simple Effects
– Between-Subjects Effects at a Fixed Level of Within-Subjects Factor
– Within-Subjects Effects at a Fixed Level of Between-Subjects Factor
– Cell Mean Comparisons
– Interaction Contrasts
Confidence Intervals for Contrasts
Assumptions of the Multivariate Approach
Multivariate and Mixed-Model Approaches for Testing Within-Subjects Contrasts
– Comparison of the Multivariate and Mixed-Model Approaches
Optional
– More Complex Designs
Summary of Main Points
Important Formulas
– Two-Way Within-Subjects Designs
– Split-Plot Designs
Online Materials Available on DesigningExperiments.com
Exercises

Part IV: Mixed Effects Models
Chapter 15: An Introduction to Mixed-Effects Models: Within-Subjects Designs
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
Advantages of Mixed-Effects Models
– Within-Subjects Designs
– Overview of Remainder of Chapter
Within-Subjects Designs
– Various Types of Within-Subjects Designs
– Models for Longitudinal Data
– Review of the ANOVA Mixed-Model Approach
Mixed-Effects Models
– A Maximum Likelihood Approach
– An Example of Maximum Likelihood Estimation
– Comparison of ANOVA and Maximum Likelihood Models
– Numerical Example
– A Closer Look at the Random Effects Model
– Graphical Representation of Longitudinal Data
– Graphical Representation of the Random Intercept Model
– Coding Random Effects Predictor Variables
– Random Effects Parameters
– Numerical Example
– Graphical Representation of a Model With Random Slope and Intercept
– Further Consideration of Competing Models
– Additional Models
– Straight-Line Change Model
– Graphical Representation of a Growth Curve Model
– Design Considerations
An Alternative Approach and Conceptualization
– Additional Covariance Matrix Structures
– Tests of Contrasts
– Overview of Broader Model Comparison
Complex Designs
– Factorial Fixed Effects
– Multiple Variables Measured Over Time
Unbalanced Designs
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Chapter 16: An Introduction to Mixed-Effects Models: Nested Designs
Overview of Chapter: Research Questions Addressed
Published Example
Introduction
Review of the ANOVA Approach
Mixed-Effects Models Analysis for the Simple Nested Design
– Numerical Example — Equal n
– Numerical Example — Unequal n
Mixed-Effects Models for Complex Nested Designs
– Hierarchical Representation of the Model for a Simple Nested Design
– Models With Additional Level 2 Variables
– Models With Additional Level 1 Variables
Summary of Main Points
Important Formulas
Online Materials Available on DesigningExperiments.com
Exercises

Appendix (Statistical Tables)
Table 1: Critical Values of t Distribution
Table 2: Critical Values of F Distribution
Table 3: Critical Values of Bonferroni F Distribution With 1 Numerator df and Familywise α of .05
Table 4: Critical Values of Studentized Range Distribution
Table 5: Critical Values of Studentized Maximum Modulus Distribution
Table 6: Critical Values of Dunnett’s Two-Tailed Test for Comparing Treatments to a Control
Table 7: Critical Values of Dunnett’s One-Tailed Test for Comparing Treatments to a Control
Table 8: Critical Values of Bryant-Paulson Generalized Studentized Range
Table 9: Critical Values of Chi-Square Distribution
Table 10: Coefficients of Orthogonal Polynomials
Table 11: Pearson-Hartley Power Charts

References 

Name Index 

Subject Index

ISBN Information

ISBN: 978-1-138-89228-6 (hbk)
ISBN: 978-1-315-64295-6 (ebk)