nickchk.com

Nick C. Huntington-Klein

Assistant Professor of Economics, Seattle University nhuntington-klein@seattleu.edu. For phone information, email me.

I am an assistant professor of economics at Seattle University. I work in econometrics, causal inference, and higher education policy. I am best-known for my widely-shared educational materials on econometrics. On this page you can find my research, as well as links to my many different resources, software packages, and projects you may find useful.

Links

• PDF version of this CV
• My introductory textbook on causal inference and research design, The Effect: An Introduction to Research Design and Causality, is now available. Please do read it for free. Or, even better, order a physical copy from Chapman & Hall/CRC or Amazon! It’s pretty good I think.
• Also check out my straightforward and concrete supplemental Principles of Micro book A Step-by-Step Guide to the Principles of Microeconomics by Kona Publishing!

Academic Publications

Please email me at nhuntington-klein@seattleu.edu for working PDFs if you cannot access these papers.

Baker, Rachel and Nick Huntington-Klein. 2025. “Defining and Measuring Task Complexity in Major Requirements. AERA Open. (11)1, p. 1-25. (Link)

Abstract: Graduating from college requires understanding major curricular requirements and making several complex interdependent choices to fulfill them. But curricular requirements can be complex and students report that it is not always clear which classes they need to take. In this paper, we create measures to describe and quantify complexity in major requirements and compare complexity across disciplines and across universities. We find wide variation. Much of this variation is not explained by stable differences across departments and across campuses, which indicates that there may be room for simplification. Academic planners, the faculty, and administrators tasked with creating and modifying curricular plans, should consider what kind of academic gauntlets they are creating and curricular complexity should be a part of planning conversations.

Grace, James, Nick Huntington-Klein, Billy Schweiger, Melinda Martinez,Michael Osland, Laura Feher, Bogdan Chivoiu, Darren Johnson, Glenn Guntenspergen, Karen Thorne. “Causal effects vs causal mechanisms: Two traditions with different requirements and contributions towards causal understanding” Ecology Letters. Forthcoming. (Link)

Abstract: The scientific aspiration of building causal knowledge has received little explicit discussion in ecology despite its fundamental importance. When methods are described as “causal”, emphasis is increasingly placed on statistical techniques for isolating associations so as to quantify causal effects. In contrast, natural scientists have historically approached the pursuit of causal knowledge through the investigation of mechanisms that interconnect the components of systems. In this paper, we first summarize a recently published multi-evidence paradigm for causal studies meant to reconcile conflicting viewpoints. We then describe some of the basic principles of causal statistics and the challenge of estimating pure causal effects. We follow that by describing basic principles related to causal mechanistic investigations, which focus on characterizing the structures and processes conveying causal effects. While causal statistics focuses on estimating effect sizes, mechanistic investigations focus on characterizing the attributes of the underlying structures and processes linking causative agents to responses. There are important differences between how one approaches each endeavor, as well as differences in what is obtained from each type of investigation. Finally, the case is made that an explicit assessment of existing mechanistic knowledge should be an initial step in causal investigations.

Marcoci, Alexandru, David P. Wilkinson, Ana Vercammon, Bonnie C. Wintle, and 51 others including Nick Huntington-Klein. “Predicting the Replicability of Social and Behavioural Science Claims in a Crisis: The COVID-19 Preprint Replication Project.” Nature: Human Behavior. Forthcoming. (Link)

Abstract: Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis. We elicited judgments from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated both their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgments on the full set of claims were correlated (r=.48). These results suggest that both beginners and more experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required into understanding who the right experts in forecasting replicability are and how their judgements ought to be elicited.

Sarafoglou, Alexandra, Suzanne Hoogeveen, Don van den Bergh, and 37 others including Nick Huntington-Klein. 2024. “Subjective Evidence Evaluation Survey For Many-Analysts Studies.” Royal Society Open Science. (Link)

Abstract: Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same data set by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g., effect size) provided by each analysis team. Although informative about the range of plausible effects in a data set, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item Subjective Evidence Evaluation Survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Jilla, Anna Marie, Johnson, Carole E., and Nick Huntington-Klein. 2024. “Benefit-Cost Analyses of Hearing Aids, Over-the-Counter Hearing Devices, and Hearing Care Services.” American Journal of Audiology. (Link)

Abstract: Limited accessibility to hearing health care threatens healthy aging in place for older adults with hearing loss. A willingness-to-pay (WTP) approach was used to determine benefits and value of prescription advanced digital technology (ADT) hearing aids, over-the-counter (OTC) hearing devices, and rehabilitative services using benefit-cost analyses.

Huntington-Klein, Nick. 2023. “Linear Rescaling to Accurately Interpret Logarithms.” Journal of Econometric Methods. 12(1): 139-147. (Link, arXiv)

Abstract: The natural logarithm transformation is an important tool in statistical analysis. The standard approach to interpreting a linear change of p in ln(X) is as a (1+p) proportional change in X. This is an approximation that is only valid for small values of p. In this paper I suggest the use of base-(1+p) logarithms (linearly rescaling ln(X) by 1/ln(1+p)), where p is chosen ahead of time. Then, a one-unit change in log1+p(X) is exactly equivalent to a (1+p) proportional change in X. This method avoids an approximation being applied outside its useful range, offers an easier way of reporting exact interpretations, improves approximation quality when approximations are used, produces a logarithm more similar to other regression variables by making the change of interest a one-log-unit change, and reduces error from the use of log(1+X) in cases where X can be 0.

Jilla, Anna Marie, Johnson, Carole E., and Nick Huntington-Klein. 2023. “Hearing Aid Affordability in the United States.” Disability and Rehabilitation: Assistive Technology. 18 (3): 246-252. (Link, Replication files.)

Abstract: Substantial out-of-pocket costs for hearing aids constitute a barrier to hearing health care accessibility for older adults among whom prevalence of hearing loss is high. This study is the first to estimate the proportion of Americans with functional hearing loss for which out-of-pocket expenditures for hearing aids would be unaffordable at current average costs and determine how affordability varies by sociodemographic factors.

Huntington-Klein, Nick. 2022. “Pearl Before Economists: The Book of Why and Empirical Economics.” Journal of Economic Methodology. (29)4: 326-344. (Link)

Abstract: Structural Causal Modeling (SCM) is an approach to causal inference closely associated with Judea Pearl and given an accessible instroduction in [Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books]. It is highly popular outside of economics, but has seen relatively little application within it. This paper briefly introduces the main concepts of SCM through the lens of whether applied economists are likely to find marginal benefit in these methods beyond standard economic approaches to causal inference. The most promising areas are those where SCM’s causal diagrams alone offer significant value: covariate selection, the development of placebo tests, causal discovery, and identification in complex models.

Huntington-Klein, Nick, Andreu Arenas, Emily Beam, Marco Bertoni, Jeffrey R. Bloem, Pralhad Burli, Naibin Chen, Paul Grieco, Godwin Ekpe, Todd Pugatch, Martin Saavedra, and Yaniv Stopnitzky. 2021. “The Influence of Hidden Researcher Decisions in Applied Microeconomics” Economic Inquiry. 59 (3): 944-960. Winner, Economic Inquiry Paper of the Year 2021. (Link, CSUF Working Paper PDF IZA Working Paper PDF GLO Working Paper PDF Replication Files)

Abstract: Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect’s sign varied as well. The standard deviation of estimates across replications was 3-4 times the mean reported standard error.

Huntington-Klein, Nick and Andrew Gill. 2021. “Semester Course Load and Student Performance.” Research in Higher Education. 62. 623-650. (Link, Working Paper PDF. Replication materials.)

Abstract: Many college students in the United States take longer than four years to complete their bachelor’s degrees. Long time-to-degree can increase higher education costs by billions. Time-to-degree can be reduced if students take more credits each term. While academic momentum theory suggests that additional credits may also improve student performance, and there is a strong positive correlation between course load and student performance, high course load may reduce time investment in each course, giving high course load a negative causal effect on performance. Concern about the negative impact of course load on performance, especially for struggling students, may lead to pushback against policies to reduce time-to-degree by increasing course load. Using longitudinal data from a regional four-year university with a high average time-to-degree, we find no evidence that high course loads have a negative impact on student grades, even for students at the low end of the performance distribution. This result is consistent with a model where students substitute time away from non-education activities when their course loads increase.

Huntington-Klein, Nick. 2021. “Human Capital vs. Signaling is Empirically Unresolvable.” Empirical Economics. 60: 2499-2531. (Link, Current Working Version. Old working paper (less math).)

Abstract: Economists offer two main explanations for the causal labor market returns to education. The first is human capital accumulation: education improves ability. The second is signaling: education allows high-ability students to distinguish themselves. A major point of interest is the relative contributions of these effects. I demonstrate the theoretical and empirical conditions necessary to identify the relative contribution of the two models. Then, I review the existing literature to evaluate whether the feasible set of empirical estimates is capable of meeting those conditions and so informing theory. Empirical evidence is capable of rejecting pure human capital and signaling models, and usually does so. I argue that, for the general question of relative contribution, necessary identification conditions are not met, and partial identification bounds are wide. Two models with different non-zero contributions of human capital and signaling cannot be empirically distinguished, limiting the usefulness of human capital vs. signaling as a framing for understanding the return to education and for policy.

Huntington-Klein, Nick. 2020. “Instruments with Heterogeneous Effects: Bias, Monotonicity, and Localness.” Journal of Causal Inference. 8 (1): 182-208. (Link, Working Paper PDF. Replication files.)

Abstract: In Instrumental Variables (IV) estimation, the effect of an instrument on an endogenous variable may vary across the sample. In this case, IV produces a local average treatment effect (LATE), and if monotonicity does not hold, then no effect of interest is identified. In this paper, I calculate the weighted average of treatment effects that is identified under general first-stage effect heterogeneity, which is generally not the average treatment effect among those affected by the instrument. I then describe a simple set of data-driven approaches to modeling variation in the effect of the instrument. These approaches identify a Super-Local Average Treatment Effect (SLATE) that weights treatment effects by the corresponding instrument effect more heavily than LATE. Even when first-stage heterogeneity is poorly modeled, these approaches considerably reduce the impact of small-sample bias compared to standard IV and unbiased weak-instrument IV methods, and can also make results more robust to violations of monotonicity. In application to a published study with a strong instrument, the preferred approach reduces error by about 20% in small (N about 1,000) subsamples, and by about 15% in larger (N about 33,000) subsamples.

Huntington-Klein, Nick, and Andrew M. Gill. 2019. “An Informational Intervention to Increase Semester Credits in College.” Series of Unsurprising Results in Economics 1: 1-17. (Link, Working Paper PDF. Replication files.)

Abstract: Increased time to college degree completion increases tuition and foregone earnings costs. Encouraging college students to take more semester credits is a low-cost way to reduce time to completion. We implemented an experimental informational intervention to increase student course loads by varying the intensity of information about the benefits of taking 15 credits per semester. We find no effect of our treatment on students’ course loads. Our null finding is of interest because of the increasing popularity of low-cost informational interventions. Uncovering null results like these is important for the design of future interventions.

Huntington-Klein, Nick. 2018. “College Choice as a Collective Decision.” Economic Inquiry 56 (2): 1202-19. (Link, Replication files. Working Paper PDF.)

Abstract: Although the choice between colleges can be thought of as being made collectively by a family, models of educational choice almost universally portray the decision as made by the student alone. Using a novel experimental method for identifying collective decision functions, I find that students have more influence than parents over the decision, but not exclusive control. Students care more than parents about classroom experience and future earnings. Ignoring the dual-agent nature of the decision can weaken predictions and lead to poorly-targeted policy designs.

Huntington-Klein, Nick, and Elizabeth Ackert. 2018. “The Long Road to Equality: A Meta-Regression Analysis of Changes in the Black Test Score Gap Over Time.” Social Science Quarterly 99 (3): 1119-33. (Link, Replication materials.)

Abstract: This study performs a meta-regression analysis of over 1,100 regressions in 165 studies to examine the relationship between African American racial status and student achievement scores in K-12 education from 1979 to 2010. The study examines time trends in the black test score gap and estimates the extent to which controls for confounding variables including socioeconomic status and schooling characteristics attenuate the size of the gap. Across the samples in the study, the absolute relationship between Black status and achievement decreased during the 1980s and early 1990s, but has been stagnant since the late 1990s. We estimate that socioeconomic status alone explains more than half of the gap, and this influence does not vary significantly over the time period of interest. Controlling for differences in school characteristics only reduces the gap slightly, but school-level factors explain an increasing proportion of the gap over time.

Huntington-Klein, Nick, and Elaina Rose. 2018. “Gender Peer Effects in a Predominantly Male Environment: Evidence from West Point.” AEA Papers and Proceedings 108: 392-95. (Link, Replication materials)

Abstract: There is considerable interest in the success of women in overwhelmingly male environments. One hypothesized determinant of success is the increased presence of other women. However, the theoretical direction of this effect is uncertain. Previous studies of heavily male contexts have had mixed results. We take advantage of random peer group assignment at West Point military academy to identify gender peer effects in the first years in which women were admitted. We find that women do significantly better when placed in companies with more women peers. The addition of one woman peer reduces the gender progression gap by half.

Goldhaber, Dan, Cyrus Grout, and Nick Huntington-Klein. 2017. “Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools.” Education Finance and Policy 12 (2): 197-223. (Link)

Abstract: Despite their widespread use, there is little academic evidence on whether applicant selection tools can improve teacher-hiring processes. We examine two screening instruments used to select classroom teachers. The screening instruments strongly predict teacher value-added in math and teacher attrition and weakly predict value-added in reading, but do not predict teacher absences. An increase of one standard deviation in screening scores is associated with an increase of about 0.06 standard deviations of student math achievement and a decrease in teacher attrition of three percentage points. These results are robust to corrections for sample selection.

Huntington-Klein, Nick. 2017. “A Method for Estimating Local Average Treatment Effects in Aggregate Data with Imperfect Assignment.” Applied Economics Letters 24 (11): 762-65. (Link)

Abstract: In some contexts, the effect of a treatment can be estimated with easily-accessible aggregate rather than individual data, using difference-in-difference estimation. However, under imperfect assignment this produces intent-to-treat estimates, which may not be the treatment effect of interest. This paper provides a method for estimating local average treatment effects using aggregate data. I also suggest a data source that allows the method to be applied when treatment rates are not recorded.

Huntington-Klein, Nick, James Cowan, and Dan Goldhaber. 2017. “Selection into Online Community College Courses and Their Effects on Persistence.” Research in Higher Education 58 (3): 244-69. (Link)

Abstract: Online courses at the college level are growing in popularity, and nearly all community colleges offer online courses (Allen & Seaman, 2015). What is the effect of the expanded availability of online curricula on persistence in the field and towards a degree? We use a model of self-selection to estimate the effect of taking an online course, using region and time variation in internet service as a source of identifying variation. Our method, as opposed to standard experimental methods, allows us to consider the effect among students who actually choose to take such courses. For the average person, taking an online course has a negative effect on the probability of taking another course in the same field and on the probability of earning a degree. The negative effect on graduation for for students who choose to take an online course is stronger than the negative effect for the average student. Community colleges must balance these results against the attractive features of online courses, and may want to consider actively targeting online courses towards those most likely to do well in them.

Huntington-Klein, Nick. 2016. ““(Un)informed College and Major Choice”: Verification in an Alternate Setting.” Economics of Education Review 53: 159-63. (Link, Replication materials)

Abstract: In their recent paper “(Un)informed College and Major Choice: Evidence from Linked Survey and Administrative Data,” Hastings, Neilson, Ramirez, & Zimmerman (2016) provide an informal costly-information model, linking family background to students’ beliefs about educational costs and benefits. They verify predictions of their model using a data set of beliefs about college institutions and majors among Chilean college applicants and students. I test some of those same predictions using a data set of beliefs about college institutions and different levels of college education among high school students in the United States. I verify their predictions, with some exceptions, supporting the use of their costly-search model.

Long, Mark C., Dan Goldhaber, and Nick Huntington-Klein. 2015. “Do Completed College Majors Respond to Changes in Wages?” Economics of Education Review 49: 1-14. (Link)

Abstract: In an analysis connecting labor market earnings to college major choices, we find statistically significant relationships between changes in wages by occupation and subsequent changes in college majors completed in related fields of college study between 1982 and 2012. College majors (defined at a detailed level) are most strongly related to wages observed three years earlier, when students were college freshmen. The responses to wages vary depending on the extent to which there is a strong mapping of majors into particular occupations. We also find that women, blacks, Hispanics, and students with low test scores are less likely to respond to wage changes. These findings have implications for policy interventions designed to align students’ major choices with labor market demand.

Huntington-Klein, Nick. 2015. “Subjective and Projected Returns to Education.” Journal of Economic Behavior and Organization 117: 10-25. (Link, Recipient of 2013 Storer Award for Labor Economics.)

Abstract: There is significant heterogeneity over high school students in the wage and employment rate returns to education. I evaluate this heterogeneity using subjective returns derived from a data set of high school juniors and seniors in Washington State. Variation over observables in projected returns estimated using observed data is uncorrelated with variation in subjective returns elicited by directly asking students about their beliefs. These results mean that returns estimated using observed data are likely a very weak proxy for student beliefs.

Software

“Streetmatrix Index”. 2025. Streetmatrix.

The Streetmatrix index uses a hedonic approach to regional home variation so as to track price movements by region.

“r2especcurve: A specification curve builder for the R2E project.” 2024. r2especcurve on GitHub.

Takes data that follows the R2E robustness reproducibility in economics protocol and makes specification curves out of it. Follows R2E protocols and offers easy graph construction in line with protocol guidelines.

“causaldata: A package with example data sets from causal inference textbooks.” 2021. causaldata on GitHub. causaldata on CRAN. causaldata in Stata. causaldata on PyPI.

This package offers data sets for running code examples from causal inference textbooks.

“did: A Stata package for running the Callaway and Sant’Anna R package did.” 2021. did.

This package offers a Stata-syntax way of using the Callaway and Sant’Anna (2020) estimator for estimating difference-in-differences with staggered treatment timing. It calls the R package did written by Callaway and Sant’Anna.

“SafeGraphR: A package for reading, processing, and normalizing SafeGraph data.” 2020. SafeGraphR.

This package is for the purpose of working with SafeGraph data. It provides functions that make it easy to read in the data, process it, normalize it, and present it.

“MagnifiedIV.” 2020. MagnifiedIV for R on GitHub, MagnifiedIV for Stata on GitHub.

These packages help to run the super-local-average-treatment-effect identifying estimators described in “Instruments with Heterogeneous Effects: Bias, Monotonicity, and Localness.” (2019). CSUF Department of Economics Working Paper 2019/006.

“suncorr: A visualization tool to create ‘sun and moon’ visualizations of correlation matrices.” 2019. suncorr on GitHub.

Sun and moon correlations offer a new way to visualize correlation matrices, by positioning each variable radially as a ‘sun’ in the solar system, surrounded by each of the other variables in the same order, with correlation direction and size given by color and moon size.

“pmdplyr: Panel Maneuvers in dplyr - An R package for cleaning and manipulating panel and hierarchical data.” 2019. pmdplyr on CRAN. pmdplyr website.

pmdplyr is an R package for the purpose of manipulating panel data. It introduces a flexible panel data object, the pibble, and functions that respect the panel structure of the pibble. All pmdplyr packages work with hierarchical data, or in other contexts where there is more than one observation per individual/time period.

“CVRoller: A Python program designed to automatically generate and update CVs.” 2018. cvroller.com.

This is a general-purpose automatic document generation language tuned specifically towards the generation of academic CVs. Multiple different CV variants can be updated at once from centrally updated CV data. CVRoller allows CV generation in HTML, PDF, and Markdown formats, and was used to make the CV you are reading right now.

“vtable: A quick and easy variable browser for R.” 2018. vtable on CRAN. vtable website.

This is an R package for the purpose of viewing information about data while working on it. vtable() automatically generates and displays a table of information about the variables in a data set, including name, class, range, labels, and summary statistics.

Other Work

Please email me at nhuntington-klein@seattleu.edu for working PDFs if you cannot access these papers.

“State ‘Free College’ Programs” with William Zumeta. 2020. Council of Independent Colleges.(Link)

Abstract: This analytical report responds to the current policy interest in some states in offering “tuition-free college” to income-eligible students attending certain public colleges and universities. In most cases only public two-year colleges are covered, although New York State’s Excelsior Scholarship program also covers public four-year colleges and universities. In Section One, we use the latest available empirical data to assess the early effects of the Excelsior program and the two longest-standing state “free community college” programs, Tennessee Promise and Oregon Promise, each of which is only a few years old. We also contrast these largely single-sector “free college” approaches to college affordability policy with the approach of another state, Washington. This state also makes a commitment to cover college costs for low-income students, as well as to give generous help to more moderate-income students, but without distorting aided students’ choices among higher education sectors (two-year vs. four-year or public vs. private). Sections Two and Three of the report analyze alternative approaches for states to create cost-effective incentives for increases in college enrollment and degree production.

“A Study of West Point Shows How Women Help Each Other Advance” with Elaina Rose. 2018. Harvard Business Review.(Link)

Abstract: This Harvard Business Review article summarizes work by myself and Elaina Rose on the performance of women at the West Point military academy in the early 1980s. We look at the progression gap between men and women, and whether women tended to do better when randomly placed in companies that have a heavier concentration of women.

“Student Courseload at CSU Fullerton” with Andrew Gill. 2018. Report for CSU Fullerton.

Abstract: This report details a pair of studies performed at CSU Fullerton. The first uses administrative data to examine the effect of an increased student courseload on student performance. The second examines an experiment performed with the goal of increasing student courseload.

“Assessing the Effects of Tuition-Free Community College in Maryland” with William Zumeta. 2017. Maryland Independent Colleges and Universities Association.

Abstract: This report examines the likely financial impact on students and the state of a potential plan for Maryland to implement tuition-free community college.

“Utilizing Independent Colleges and Universities to Fulfill States’ College Degree Attainment Goals” with William Zumeta. 2017. The Council of Independent Colleges.(Link)

Abstract: America’s diverse higher education landscape includes more than 700 four-year nonprofit colleges and universities that focus on baccalaureate education. These private nondoctoral (PND) institutions are located in almost every state and collectively enroll about 1.6 million students and award nearly 150,000 degrees annually, with the majority of these being bachelor’s degrees. As this report will show, these independent colleges and universities are effective and efficient academic enterprises and, as such, are a valuable resource to the states in which they are located, as well as to the nation.

“The Cost-Effectiveness of Undergraduate Education at Private Nondoctoral Colleges and Universities” with William Zumeta. 2015. The Council of Independent Colleges.(Link)

Abstract: This study examines key aspects of the cost-effectiveness of PND colleges as providers of baccalaureate degrees and explores how states might feasibly make better use of these colleges to produce more degrees efficiently. The study looks at degree production and cost in the PND sector relative to other higher education sectors, focusing on the most comparable public institutions. PND colleges and universities have a 22 percentage point edge over comparable public institutions in four-year graduation rates and a nearly 12 point advantage in six-year graduation rates, and they hold a significant advantage for all subgroups. Moreover, PND colleges retain students initially interested in STEM and health to degrees in those majors at rates (41 percent) approaching twice the rates of public doctoral and nondoctoral institutions (24 and 23 percent, respectively).

“Understanding Sub-baccalaureate Certificate Production and Incidence in Washington’s Labor Force Through 2023” with William Zumeta. 2015. Washington Student Achievement Council.

Abstract: This report describes the production of sub-baccalaureate certificates in Washington State, and projects the growth of certificate production through 2023.

“It’s Selective, But Is It Effective? Exploring the Predictive Validity of Teacher Selection Tools” with Dan Goldhaber and Cyrus Grout. 2014. CEDR Policy Brief 2014-9.

Abstract: Evidence suggests teacher hiring in public schools is ad-hoc and often does not result in good selection amongst applicants. Some districts use structured selection instruments in the hiring process, but we know little about the efficacy of such tools. In this paper we evaluate the ability of applicant selection tools used by the Spokane Public Schools (SPS) to predict three outcomes: measures of teachers’ value-added contributions to student learning, teacher absence behavior, and attrition rates. We observe all applicants to the district, both those who are and who are not hired. We find that the screening instruments predict teacher value-added in student achievement and teacher attrition, but not teacher absences.

Working Papers

Please email me at nhuntington-klein@seattleu.edu for working PDFs if you cannot access these papers.

“The Sources of Researcher Variation in Economics” with Claus Portner and 148 others. IZA, I4R, SSRN, 2025.(SSRN Link. I4R Link. IZA Link.)

Abstract: We use a rigorous three-stage many-analysts design to assess how different researcher decisions—specifically data cleaning, research design, and the interpretation of a policy question—affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.

“Do LLMs Act as Repositories of Causal Knowledge?” with Eleanor J. Murray. arXiV. 2024.(Link)

Abstract: Large language models (LLMs) offer the potential to automate a large number of tasks that previously have not been possible to automate, including some in science. There is considerable interest in whether LLMs can automate the process of causal inference by providing the information about causal links necessary to build a structural model. We use the case of confounding in the Coronary Drug Project (CDP), for which there are several studies listing expert-selected confounders that can serve as a ground truth. LLMs exhibit mediocre performance in identifying confounders in this setting, even though text about the ground truth is in their training data. Variables that experts identify as confounders are only slightly more likely to be labeled as confounders by LLMs compared to variables that experts consider non-confounders. Further, LLM judgment on confounder status is highly inconsistent across models, prompts, and irrelevant concerns like multiple-choice option ordering. LLMs do not yet have the ability to automate the reporting of causal links.

“A Case Study of Preregistered Confirmation” with Veli-Matti Karhulahti. PsyArXiV. 2024.(Link)

Abstract: In 2019, China limited local minors’ videogame playtime to 1.5 hours on weekdays and 3 hours on public holidays. This mandate became a topic of global debate especially after it was further tightened in 2021 and gaming was completely denied from minors on weekdays. Following the above, Zendle et al. designed a study to test the effectiveness of the 2019 mandate. The exceptional data obtainedby Zendle et al. consist of billions of gaming hours and profiles, thus providing unique access to potential knowledge regarding the shifting trends in perhaps the currently largest segment of commercial digital entertainment. The study should be commended for opening new research avenues on global technology use. They concluded no evidence for the effectiveness of regulation. The preregistered design was unable to testits hypothesis for two main reasons. First, because the effect of the regulation cannot be larger than the regulated—authors entertain 6.4% of the data being minors—the test would falsify regulation success even if all minors complied. As the preregistered meaningful effect was larger than the intervention, the findings were fixed pre-data. Second, the design (Fisher’s exact test) for comparing pre-regulation (0.77%) and post-regulation (0.88%) heavy play was unable to consider other variables due to which the observed changes were larger than perfect regulation could ever produce. Had the study useda realistic effect, the design would not have been able to test it anyway

“The Changing Importance of Earnings in College Major Choice” with Elizabeth Ackert. CSU Fullerton Department of Economics Working Paper 2018/006.(PDF)

Abstract: Prior studies find that undergraduate major choices are responsive to earnings associated with those majors, but weakly suggests that responsiveness has dropped over time. Using data on college graduates from 1973 to 2013, we find that responsiveness of major choice to labor market returns has weakened over time. The weakening response is due to changes within demographic groups rather than demographic changes in the college graduate population over time. If the goal is to maintain or increase the alignment between college major and labor market returns, incentivizing undergraduates to select high-earning majors is necessary.

“Student Preference for Guidance and Complexity in College Major Requirements” with Rachel Baker. CEPA Working Paper 04/2018 and CSU Fullerton Department of Economics Working Paper 2018/005.(PDF)

Abstract: In order to graduate with a bachelor’s degree, students must determine which classes they must take in order to satisfy the requirements of their major. These requirements are often complex and difficult to comprehend, leading to some policy interventions that aim to reduce complexity by either increasing the amount of student guidance in course choice or by reducing the amount of complexity-increasing choice. We perform two student preference experiments on students at two large four-year universities to determine how students might respond to increasing guidance or reduced choice in their course-taking options. We find that students do not respond strongly to increases in guidance such as grouping courses into meaningful categories or removing cross-cutting requirements, but strongly reject a reduction in options, even when given a rationale for the reduction. These results suggest that increased-guidance policies have some avenues to operate in without student push-back, but that strong reductions in choice are unlikely to be popular.

Dormant Working Papers

The File Drawer

Writing in Broader Media

A Smarter Way to Use the Strengths of your Instrumental Variables. Nick Huntington-Klein, Causal Science, April 27, 2021.

47 Million Americans Expect to Miss a Credit Card Due Date in 2021. John S. Kiernan, Feb. 16, 2021, WalletHub.

Ask the Experts: Cheap Car Insurance. Jan. 25, 2021, WalletHub.

I am responsible for a number of entries in the SafeGraph Blog.

I am the founder of the Library of Statistical Techniques and am responsible for many of its entries.

Media Appearances, Interviews, and Mentions

How the Education Department Shutdown Impacts Washington State, KING 5 News, Feb 10, 2025.

Analyzing the Analysts: Reproducibility with Nick Huntington-Klein. Casual Inference, April 3, 2024.

Mad Science. Bandit Theater, Feb 3, 2023.

What’s Going on With the FAFSA?. The Spectator, Nov. 2, 2023.

The Collapse of Silicon Valley Bank, KIRO 7 TV News, March 15, 2023.

How much of the $754.6 million Powerball jackpot does the winner actually get to take home? KING 5 News. March 3, 2023.

Ticketmaster Torpedos Taylor Tour. The Spectator, Nov. 30, 2022.

The Effect. New Books Network Podcast: Economics, Aug. 4, 2022.

Interview with Dr. Huntington-Klein. Scholarly Spark Podcast Episodes 8-14, Oct. 29, 2021.

What Happens When Researchers “Clean” Data?. Edward Hearn, BuiltIn, June 4, 2021.

This Century’s Roaring 20s Includes Changes for Retail and Restaurants. Robin Rothman. Colorado Real Estate Journal, May 2021.

One Study, Many Results Matt Clancy, New Things Under the Sun, May 4, 2021.

The Influence of Hidden Researcher Decisions in Applied Microeconomics Tyler Cowen, Marginal Revolution, March 29, 2021.

A Replication Crisis in the Making? Jorg Peters, Elephant in the Lab, March 22, 2021.

Current State of the Stimulus Package. Kali Herbst Minino, The Spectator, February 18, 2021.

We’ve been cooped up with our families for almost a year. This is the result.. Andrew Van Dam, Feb. 16, 2021, Washington Post.

eigenrobot vs. nickchk The Eigenfriends Podcast, Feb. 10, 2021.

Hudson Yards Is Open For Business - but Who’s Coming?. Michael Herzenberg, January 31, 2021. NY1.

Black Friday Foot Traffic in 2019 vs 2020. December 4, 2020. SafeGraph Blog.

Rural Americans Stopped Staying In. Then Covid-19 Hit. Andrea Fuller and Tawnell D. Hobbs. Nov. 24, 2020, Wall Street Journal.

Gambling, gardening, and keeping the lights on: cell phone data reveals where Mid-Southerners go. Jeni Diprizio. ABC Local 24. November 6, 2020.

Foot Traffic Still Down More than 90% in Seven NYC Zip Codes. New York Spectrum 1 News, September 25, 2020.

CIC Report Reviews States’ Tuition-Free College Programs. Inside Higher Ed, September 1, 2020.

Anything But Dismal: Online K-12 Education. Anything But Dismal, July 26, 2020.

My (usually uncredited) work with SafeGraph has appeared in a long list of media outlets. A subset includes multiple times on CNN and Fox News as well as in text outlets including Bloomberg, New York Times, and NPR.

As States Start to Reopen, Here’s Where People are Going. Bonnie Berkowitz and Kevin Schaul, Washington Post, May 29, 2020.

Job Market Slows As a New Decade Begins, According to Bureau of Labor Statistics Data. Daniel Coats, Mihaylo BizBlog, January 23, 2020.

IPA’s Weekly Links. Chris Blattman, October 26, 2019.

Congress must address gender gap in nominations to military service academies. The Hill, Liam Brennan, August 18, 2019.

New journal provides outlet for research not deemed sexy enough. Stuff, June 6, 2019.

New academic journal only publishes ‘unsurprising’ research rejected by others. CBC Radio, May 27, 2019.

The faiV: Week of May 17, 2019: The Causality Edition. Fai, May 17, 2019

This Economics Journal Only Publishes Results that are No Big Deal. Vox, May 17, 2019

Inflation Raised CSU Prices. The Daily Titan, May 10, 2019

These Open Resources Will Help You Master Statistics. Forbes, February 28, 2019

Study of West Point Cadets Shows How Women Help Each Other Advance. CSU Fullerton News Center, January 25, 2019

Chico’s, Unilever GSK, ‘Handmaid’s Tale’ Sequel: Broadsheet November 29. Fortune Magazine, December 1, 2018

The faiV: Week of November 26, 2018: The Astounding Edition. fai, November 30, 2018

Do Women Actually Help Other Women Succeed? Ask the Army. Fortune, Claire Zillman, November 29, 2018.

High-Five Your Work Wife: New Study Shows How Women Help Each Other Advance. PureWow, Sarah Stiefvater, November 29, 2018

On Caplan Educational Signaling (#6). Tuesday Assorted Links, Marginal Revolution, November 27, 2018

Killing Colleges in Massachusetts. Inside Higher Ed, August 29, 2018

Cal State Fullerton professor wonders whether major course requirements could be made simpler. The Daily Titan, February 26, 2018

Private colleges can partner to solve issues. The Edwardsville Intelligencer, May 8, 2017

Findings on Education Finance and Policy Discussed by Investigators at University of Washington. Education Letter, May 3 2017

The Value of a College Degree. The Inter-Mountain, April 27, 2017

Incentives to Attend Private Colleges Could Save States Money and Raise Graduation Rates. The Chronicle of Higher Education, April 14, 2017

Hiring Successful Teachers - Two New Studies Point to What Works and What Doesn’t. EdSurge, October 3, 2016

CSUF researcher studies how stereotypes affect higher education decisions. OC Register, May 12, 2016

How Will Your College Degree Pay Off? Mihaylo College Bizblog, January 4, 2016

Facing growing scrutiny, colleges set out to prove their value. The Hechinger Report, January 22, 2016.

Is Robert anti-teacher? The Education Gadfly, November 5, 2014

Study: Teacher hiring should be more scientific. Associated Press, October 29, 2014