p-hacking Concepts

Cult of the Dead Cow: How the Original Hacking Supergroup Might Just Save the World

by Joseph Menn · 3 Jun 2019 · 302pp · 85,877 words

SQL Hacks

by Andrew Cumming and Gordon Russell · 28 Nov 2006 · 696pp · 111,976 words

The Ethical Algorithm: The Science of Socially Aware Algorithm Design

by Michael Kearns and Aaron Roth · 3 Oct 2019

performing the same experiment, or repeatedly running different statistical tests on the same dataset, but then only reporting the most interesting results is known as p-hacking. It is a technique that scientists can use (deliberately or unconsciously) to try to get their results to appear more significant (remember from the beginning

…

work—are not the sorts of things that will get published in high-prestige venues. The result is that even in the absence of explicit p-hacking by individual researchers or teams, published papers represent an extremely skewed subset of the research that has been performed in aggregate. We see reports of

…

(each in good faith), but only the one with the most surprising result ends up being published. The Sport of Machine Learning The dangers of p-hacking are by no means limited to the traditional sciences: they extend to machine learning as well. Sandy Pentland, a professor at MIT, was quoted in

…

apparently don’t get out of the office enough to have seen real trees.) Fig. 26. Tree illustrating the dangers of adaptive data analysis and p-hacking. Each level of the tree corresponds to a feature that could be correlated (left) or anti-correlated (right) with the label. The gray path (LRRL

…

been viewed with skepticism, going by various derogatory names including “data snooping,” “data dredging,” and—as we have seen—“p-hacking.” The methodological dangers presented by the combination of algorithmic and human p-hacking have generated acrimonious controversies and hand-wringing over scientific findings that don’t reflect reality. These play a central role

…

themselves. The crisis has long-standing roots; the phrase was coined in the early 2010s as part of a growing awareness of the problem. While p-hacking is not the only culprit here—poor study design, sloppy experimental technique, and even occasionally outright fraud and deception are present—the concern is that

…

scientific exploration can lead to false findings that cannot be reproduced on fresh data or in fresh experiments. Among the more prominent examples associated with p-hacking was the controversial research on “power poses” that we discussed earlier. Here is how it was described in the New York Times Magazine in late

…

the study failed to be replicated, and “power poses” became the poster child for the reproducibility crisis and the dangers of p-hacking. “We realized entire literatures could be false positives,” [prominent p-hacking critic Joe] Simmons says. “They had collaborated with enough other researchers to recognize that the practice was widespread and counted

…

guilty.” The famous priming studies we discussed also failed to hold up to scrutiny. And food science has been under suspicion for years. A notable p-hacking scandal rocked the food science community in 2017. In this instance the principal researcher in question, a celebrated Cornell professor named Brian Wansink, seemed to

…

actively embrace p-hacking as a means of generating results. The following is a description of instructions he sent to a student working with him, whom he wanted to

…

on the website of the Open Science Foundation, one of the most prominent organizations providing preregistration of datasets, hypotheses, and analyses. To further guard against p-hacking, preregistrations cannot be deleted. Fortunately, a recent set of algorithmic advances points to a solution: a methodology to safely reuse data, so that we avoid

…

rich and useful theory, and privacy and fairness are off to fast starts in this regard. Similarly, our chapters on algorithmic game theory and preventing p-hacking were partly chosen for the relative maturity of the research in those fields. But what about other values, such as having algorithms and models that

…

, 74–84, 87, 192–93 impact of ethical behaviors, 19 ACT tests, 65 adaptive data analysis, 139, 143–45, 151–53, 159, 166. See also p-hacking advantages of machine learning, 190–93 advertising, 191–92 Afghanistan, 50–51 age data, 27–29, 65–66, 86–89 aggregate data, 2, 30–34

…

fairness, 85 and ethical issues of optimization, 189–90 and investing scams, 140–41 and medical residency hiring, 128 and navigation problems, 111–15 and p-hacking, 144–45 and scientific research, 136, 144–45 and user preferences, 97 income discrimination, 88–89 incriminating information, 40–45 individual preferences, 115–17 inference

…

and Society, 15 patient records. See medical records and data Pentland, Sandy, 145–46 performance reporting, 140–41 personal data, 171 personalized medicine, 191–92 p-hacking, 144–46, 153–59, 161, 169–70. See also adaptive data analysis phone apps, 2 physiology research, 141–43 plausible deniability, 41 policy, 82–84

…

reasoning, 140–41 and interpretability of model outputs, 171–72 and investing scams, 138–41 and medical research, 34 and online shopping algorithms, 117 and p-hacking, 144–45, 153–55, 157–59, 161, 164, 169–70 statistical modeling, 90 statistical parity, 69–74, 84 and US Census data, 195 and “word

New Dark Age: Technology and the End of the Future

by James Bridle · 18 Jun 2018 · 301pp · 85,263 words

are sexed up and counterexamples quietly filed away – to the very tools with which scientific results are generated. The most controversial of these techniques is p-hacking. P stands for probability, denoting the value at which an experimental result can be considered statistically significant. The ability to calculate a p-value in

…

particular goal to aim for, can selectively cull from great fields of data in order to prove any particular hypothesis. As an example of how p-hacking works, let’s hypothesise that green dice, uniquely among all other dice, are loaded. Take ten green dice and roll each of them one hundred

…

it’s easy to read, meaning that more and more journals use it as shorthand for reliability when sifting through potentially thousands of submissions. Moreover, p-hacking doesn’t just depend on getting those serendipitous results and running with them. Instead, researchers can comb through vast amounts of data to find the

…

I rolled, the more likely I would be to get an anomalous result – and this is the one I could publish. This practice has given p-hacking another name: data dredging. Data dredging has become particularly notorious in the social sciences, where social media and other sources of big behavioural data have

…

suddenly and vastly increased the amount of information available to researchers. But the pervasiveness of p-hacking isn’t limited to the social sciences. A comprehensive analysis of 100,000 open access papers in 2015 found evidence of

…

p-hacking across multiple disciplines.16 The researchers mined the papers for every p-value they could find, and they discovered that the vast majority just scraped

…

’, Nature, May 25, 2016, nature.com. 15.For more on the math of this experiment, see Jean-Francois Puget, ‘Green dice are loaded (welcome to p-hacking)’, IBM developer-Works blog entry, March 22, 2016, ibm.com. 16.M. L. Head, et al., ‘The Extent and Consequences of

…

P-Hacking in Science’, PLOS Biology 13:3 (2015). 17.John P. A. Ioannidis, ‘Why Most Published Research Findings Are False’, PLOS ONE, August 2005. 18.Derek

…

,’ 23–4 Bush Differential Analyser, 27 on hypertext, 79 Bush Differential Analyser, 27 Byron “Darkness,” 201–2 C Cadwalladr, Carole, 236 calculating machines, 27 calculation p-hacking, 89–91 raw computing, 82–3 replicability, 88–9 translation algorithms, 84 Cambridge Analytica, 236 Campbell, Duncan, 189 ‘Can We Survive Technology?’ (von Neumann), 28

…

), 25–6 P Paglen, Trevor, 144 ‘paranoid style,’ 205–6 Patriot Act, 178 Penrose, Roger, 20 Perceptron, 136–8, 137 permafrost, 47–9, 56–7 p-hacking, 89–91 Phillippi, Harriet Ann, 165 photophone, 19–20 Pichai, Sundar, 139 Piketty, Thomas Capital in the Twenty-First Century, 112 Pincher, Chapman, 175–6

Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth

by Stuart Ritchie · 20 Jul 2020

’, Wansink had unintentionally revealed a major flaw in the way he, and unfortunately many thousands of other scientists, conducted research. That flaw has been dubbed ‘p-hacking’.47 Because the p < 0.05 criterion is so important for getting papers published – after all, it supposedly signals a real effect – scientists whose

…

studies show ambiguous or disappointing results regularly use practices that ever so slightly nudge, or hack, their p-values below that crucial threshold. Such p-hacking comes in two main flavours. In one, scientists pursuing a particular hypothesis run and re-run and re-re-run their analysis of an experiment

…

scientist can then declare, often perhaps convincing even themselves, that they’d been searching for these results from the start.49 This latter type of p-hacking is known as HARKing, or Hypothesising After the Results are Known. It’s nicely summed up by the oft-repeated analogy of the ‘Texas

…

the few bullet holes that happen to be near to one another, claiming that’s where he was aiming all along.50 Both kinds of p-hacking are instances of the same mistake and ironically, it’s precisely the one that p-values were invented to avoid: capitalising on random chance.

…

the p < 0.05 criterion, you might still get misled by random chance. But this risk is substantially lower than in the case of p-hacking, where running multiple tests multiplies the risk that any one of them is misleading. That’s the fundamental insight that so many scientists don’t

…

own foot. After Wansink’s blog post appeared, some sceptical readers started digging into the numbers in his papers.55 It turned out that the p-hacking was only one of a multitude of statistical screw-ups. Across the four papers he’d published using the pizza dataset, the team of

…

As outrageous as they were, all of Wansink’s errors and blunders were a distraction from the most widely applicable lesson of the affair: his p-hacking. Amid the media scandal that arose when the retractions started rolling in, a BuzzFeed News journalist published an incriminating and highly revealing email from Wansink

…

desire to ‘get that one value below .05’ is strong – and, given journals’ clear preference for exciting, flashy, positive results, that’s definitely the case – p-hacking is the near-inevitable result. Wansink’s accidental confession ended badly for him. However, when another prominent scientist was forthright in admitting that they’d

…

’.64 I couldn’t find a single negative reaction – except the one from Amy Cuddy herself, who took the opportunity to distance herself from the p-hacking: ‘I … cannot contest the first author’s [that is, Carney’s] recollections of how the data were collected and analyzed, as she led both

…

undermined the study on power posing? In 2012, a poll of over 2,000 psychologists asked them if they had engaged in a range of p-hacking practices.66 Had they ever collected data on several different outcomes but not reported all of them? Approximately 65 per cent said they had.

…

you ended up with, having followed your unique combination of forking paths, wasn’t a statistical fluke? Even without the trial-and-error of classic p-hacking, then, scientists who don’t come to their data with a proper plan can end up analysing themselves into an unreplicable corner. Why unreplicable?

…

to p < 0.05 in that dataset, but that won’t necessarily do the same in others. This is the trouble with all kinds of p-hacking, whether explicit or otherwise: they cause the analysis to – using the technical term – overfit the data.73 In other words, the analysis might describe

…

a major goal in science is to convince other scientists that your model, theory or study is worth taking seriously. The same motivation exists for p-hacking more generally: studies that aren’t marred by the occasional non-significant result amongst all their sub-0.05 p-values seem far more compelling

…

equally well-conducted paper that reports the outcome, warts and all, to reach a more qualified conclusion.’74 Here we see how publication bias and p-hacking are two manifestations of the same phenomenon: a desire to erase results that don’t fit well with a preconceived theory. This phenomenon was

…

from the trial are being analysed. Texas sharpshooter-style behaviour in medical trials is often called ‘outcome-switching’ (another name for what amounts to p-hacking). Let’s go back to our example of running a study of height differences between men and women. Maybe you also happened to measure some

…

research is often missing due to publication bias, if the studies included in the meta-analysis are all themselves exaggerated by p-hacking, the overall combined effect – in what’s supposed to be an authoritative overview of all the knowledge on the topic – will end up far

…

their study? Only 0.7 per cent did. There are two reasons why this is disappointing. The first is something we’ve seen before: p-hacking. Not setting the sample size beforehand allows the researchers to continue collecting data and testing it, collecting data and testing it, again and again in

…

simply hadn’t noticed all the problems. It’s safe to say that a substantial portion of media-hyped nutritional studies are also affected by p-hacking. Because so many large datasets exist with so many relevant variables – it’s typical in nutrition research for participants to fill in a so-called

…

practice and ask whether they reward objectivity – or something else entirely. So far in the book, we’ve seen scientists fabricating data, file-drawering and p-hacking their studies, failing to check for errors, and exaggerating their results. Taken together, we have a picture of scientific practice that’s fundamentally at odds

…

are scientists so deluded by their own theories, or who desire so strongly to feel they’ve made a difference, that they use fraud or p-hacking to vanish any bothersome ambiguities. There are scientists driven primarily by a desire for money, prestige, power, or fame, and who care about the

…

for salami-slicing to become a norm. Incentivise publication in high-impact journals, and you’ll get it – but be prepared for scientists to use p-hacking, publication bias and even fraud in their attempts to get there. Incentivise competing for grant money, and you’ll get it – but be prepared

…

journals don’t favour null results – but for a clear-eyed look at the scientific record, we need to see them. The second step was p-hacking – specifically, outcome-switching, where scientists change the focus of their study upon finding that their main result isn’t statistically significant. Once the outcomes

…

scientists to publish replications and null results might reduce publication bias. But what about the other forms of bias we encountered, having to do with p-hacking? Many dozens of papers, and even entire books, have been written on the pitfalls of p-values: they’re hard to understand, they don

…

their method would likely reduce, is a more pressing concern than that of false negatives. Here’s another way to deal with statistical bias and p-hacking: take the analysis completely out of the researchers’ hands. In this scenario, upon collecting their data, scientists would hand them over for analysis to

…

these tech panics comes from big observational studies looking at correlations between screen time and mental health problems in adolescents. Given the strong potential for p-hacking in such studies (remember how easy that was to do in the big datasets from nutritional research, where essentially all foods could be linked to

…

posting a plan for your analysis somewhere public, you lash yourself to the mast and stop yourself giving in to the Siren’s call of p-hacking. Some would rightly object that if scientists permit themselves no wiggle room whatsoever, there’s no longer an opportunity for serendipitous findings (penicillin and

…

kill publication bias stone dead, by removing the pernicious link between the statistical significance of the results and the decision to publish, but it reduces p-hacking as well, since you have to agree to your analysis with the reviewers beforehand and can’t just alter it post hoc without making it

…

place, since it would take the brassiest of brass necks to post a fake dataset on a public website.55 The same principle works for p-hacking that borders on fraud, and for more innocent errors: allowing other scientists to see your data and how you analysed it means that eagle-

…

(by our familiar scientific fraudsters, who have no intention of revealing their deceptions), ‘trimming’, and ‘cooking’ (both of which correspond, in our modern understanding, to p-hacking, where scientists manipulate their data and observations to give them the appearance of greater interest or accuracy). So although our modern publication system exacerbates science

…

can find an online registration of the study, it should at least modestly increase your confidence that the results aren’t just due to p-hacking.3 Tracking down the pre-registration document can also help you spot whether the main analysis is different from the one the scientists pre-registered

…

to make the 0.05 threshold. A perfect line-up of significant results in a study with many p-values is likely the result of p-hacking (or worse). 6. Are the inferences appropriate? As we’ve seen, scientists regularly slip into causal-sounding language even if they’ve only run

…

– situations where the variables involved in each test aren’t related to each other at all. In practice, and especially in many cases of p-hacking, where the same variables are being used over and over again, the increase in the false-positive rate as a function of the number of

…

’, The 20% Statistician, 14 Feb. 2016; https://daniellakens.blogspot.com/2016/02/why-you-dont-need-to-adjust-you-alpha.html 53. This analogy for p-hacking is from Lee McIntyre, The Scientific Attitude: Defending Science from Denial, Fraud, and Pseudoscience (Cambridge, Massachusetts: The MIT Press, 2019). 54. This is hardly

…

Wansink Turned Shoddy Data Into Viral Studies About How We Eat’, BuzzFeed News, 25 Feb. 2018; https://www.buzzfeednews.com/article/stephaniemlee/brian-wansink-cornell-p-hacking 62. Alas, in many cases they aren’t delicate about making their requests at all, as the researchers behind the Bullied into Bad Science initiative

…

‘The Garden of Forking Paths’, Labyrinths, tr. Donald A. Yates (New York: New Directions, 1962, 1964). 73. This framing of the p-hacking problem is due to Yarkoni & Westfall, who call p-hacking ‘procedural overfitting’: Yarkoni & Westfall, ‘Choosing Prediction’, p. 1103. 74. Roger Giner-Sorolla, ‘Science or Art? How Aesthetic Standards Grease the Way

…

Politics and Policy 8, no. 1, 26 Jan. 2017; https://doi.org/10.1515/spp-2016-0006 41. Joseph P. Simmons et al., ‘Life after P-Hacking: Meeting of the Society for Personality and Social Psychology’, SSRN, (New Orleans, LA, 17–19 Jan. 2013); https://doi.org/10.2139/ssrn.2205186 42

…

also remember the case of Amy Cuddy, whose bestselling book was based on a study that was later revealed to be a paradigm example of p-hacking. See Chapters 2 and 4. 46. Matthew Walker, Why We Sleep: The New Science of Sleep and Dreams (London: Allen Lane, 2017). 47. https

…

Labs Bem, Daryl benzodiazepines bias blinding and conflict of interest De Vries’ study (2018) funding and groupthink and meaning well bias Morton’s skull studies p-hacking politics and publication bias randomisation and sexism and Bik, Elisabeth Bill & Melinda Gates Foundation Biomaterials biology amyloid cascade hypothesis Bik’s fake images study (2016

Fluke: Chance, Chaos, and Why Everything We Do Matters

by Brian Klaas · 23 Jan 2024 · 250pp · 96,870 words

advancement. When researchers tweak their data analysis to produce a P value that’s low enough for an article to be published, that’s called P-hacking, and it’s a scourge of modern research, one that causes us to misunderstand our world. But how widespread is it? In one analysis of

…

evidence that published research is being skewed by this system. The replication crisis, partially sparked by Bem’s discredited ESP studies, blew the lid off P-hacking. Unfortunately, it didn’t do much to stop it. When economists examined data in twenty-five top economics journals many years after the replication crisis

…

, they found that up to a quarter of results using certain kinds of research methods showed misleading data interpretations and potential evidence of P-hacking. That’s a big proportion of research that affects how we see the world—and our place within it. These bogus studies, which often trace

…

straightforward causes and effects, incorrectly reinforce the notion that we can write out society’s flukes because reality—when contorted by P-hacking—does appear tidier and more ordered. X causes Y in a straightforward way, and we’ve got the low P value to prove it! Bad

…

did in the past because of important advances in the research fields that study ourselves. Social science graduate students are warned about the perils of P-hacking, and some journals are making wise efforts to address the file drawer problem. Transparency has substantially increased. Just because economists or political scientists get it

…

73 (1) (2019): 235–45. “they will confess”: Smith, “How Shoddy Data.” quarter of results: A. Brodeur, N. Cook, and A. G. Heyes, “Methods Matter: P-hacking and Causal Inference in Economics,” IZA Discussion Paper no. 11796, 2018. file drawer problem: For a scathing indictment of modern research methods, see J. P

…

and, 88 deterministic universe and, 221, 225 Holy Grail of Causality, 216 multiple causes, 89 narrative bias and, 137, 138 pattern detection and, 70, 71 P-hacking and, 201 playing-God thought experiment and, 51 reverse causality, 88 storytelling and, 73–74 superstition and, 75 timing and, 193 cDa29 (human tetrachromat), 69

…

, 197, 203, 205, 217 file drawer problem, 201, 202 Fragile Families Challenge, 216–17 Holy Grail of Causality, 216 misuse of mathematics in, 213–14 P-hacking and, 199–201, 203 replication crisis, 198–99, 200 sociology, 49, 88, 108, 196, 199 space-time, 143 spiritualism, 171 split-brain experiments, 74 stability

Data Science from Scratch: First Principles with Python

by Joel Grus · 13 Apr 2015 · 579pp · 76,657 words

interval. (The “fair coin” hypothesis doesn’t pass a test that you’d expect it to pass 95% of the time if it were true.) P-hacking A procedure that erroneously rejects the null hypothesis only 5% of the time will — by definition — 5% of the time erroneously reject the null hypothesis

…

, and you can probably get your p value below 0.05. (We did something vaguely similar in “Correlation”; did you notice?) This is sometimes called P-hacking and is in some ways a consequence of the “inference from p-values framework.” A good article criticizing this approach is “The Earth Is Round

…

, Statistical Hypothesis Testingexample, an A/B test, Example: Running an A/B Test example, flipping a coin, Example: Flipping a Coin-Example: Flipping a Coin p-hacking, P-hacking regression coefficients, Standard Errors of Regression Coefficients-Standard Errors of Regression Coefficients using confidence intervals, Confidence Intervals using p-values, Example: Flipping a Coin I

…

, NumPy O one-sided tests, Example: Flipping a Coin ORDER BY statement (SQL), ORDER BY overfitting, Overfitting and Underfitting, The Bias-Variance Trade-off P p-hacking, P-hacking p-values, Example: Flipping a Coin PageRank algorithm, Directed Graphs and PageRank paid accounts, predicting, Paid Accounts pandas, For Further Exploration, For Further Exploration, pandas

…

Continuous Distributions The Normal Distribution The Central Limit Theorem For Further Exploration 7. Hypothesis and Inference Statistical Hypothesis Testing Example: Flipping a Coin Confidence Intervals P-hacking Example: Running an A/B Test Bayesian Inference For Further Exploration 8. Gradient Descent The Idea Behind Gradient Descent Estimating the Gradient Using the Gradient

Calling Bullshit: The Art of Scepticism in a Data-Driven World

by Jevin D. West and Carl T. Bergstrom · 3 Aug 2020

The important thing to remember is that a very unlikely hypothesis remains unlikely even after someone obtains experimental results with a very low p-value. P-HACKING AND PUBLICATION BIAS Purely as a matter of convention, we often use a p-value of 0.05 as a cutoff for saying that a

…

tests until they find a way to nudge their p-values across that critical p = 0.05 threshold of statistical significance. This is known as p-hacking, and it’s a serious problem. Or they will alter the outcomes they are testing. A clinical trial may set out to measure the effect

…

published literature is likely to be representative of the set of all experiments conducted, and when, instead, the published literature reflects problematic practices such as p-hacking or publication bias. Figuring out how best to do this has become a hot area in statistics research. CLICKBAIT SCIENCE Members of the general public

…

a one-in-twenty chance that any of the tested hypotheses would appear significant if the null hypothesis were true. *9 To illustrate how powerful p-hacking techniques can be, Joseph Simmons and colleagues Leif Nelson and Uri Simonsohn tested a pair of hypotheses they were pretty sure were untrue. One was

…

are. Volunteers listened to either a children’s song or a control song, and later were asked how old they felt. With a bit of p-hacking, the researchers concluded that listening to a children’s song makes people feel older, with statistical significance at the p < 0.05 level. While suggestive

…

, the initial study was not the most persuasive demonstration of how p-hacking can mislead. Maybe listening to a children’s song really does make you feel old. So the authors raised the bar and tested a hypothesis

…

additional pieces of personal information, which they ended up discarding because they did not give the desired result. The paper makes a compelling case. If p-hacking can reverse the flow of time, what can it not do? *10 This is not a new insight. The statistician Theodore Sterling observed in 1959

On the Edge: The Art of Risking Everything

by Nate Silver · 12 Aug 2024 · 848pp · 227,015 words

of Ockham, that simpler solutions are more likely to be true. Generally regarded highly in the River, because more complex solutions can give rise to p-hacking and overfitting, where the data is tortured to produce the desired conclusion. A related term is parsimony. Odds: Sometimes a synonym for probability, but more

…

by right-wing groups in the mid-2010s, the creator of Pepe the Frog has disavowed that connotation and the meme is typically apolitical today. p-hacking: Any of a number of dubious methods to obtain an ostensibly statistically significant result to increase the chances for publication in an academic journal. The

…

Peabody, Rufus, 178–80, 181, 182–83, 191, 193, 195, 204, 517n Pepe, 492 Perkins, Bill, 374–75 Persinger, LoriAnn, 118 Petrov, Stanislav, 424, 426 p-hacking, 492 physical risk-takers, 217–21 Piper, Kelsey, 505n pips, 492 pit boss, 493 pits, 493 See also table games plurality, 470–71, 493 plus

Super Thinking: The Big Book of Mental Models

by Gabriel Weinberg and Lauren McCann · 17 Jun 2019

only one statistical test conducted. The act of running additional tests to look for statistically significant results has many names, including data dredging, fishing, and p-hacking (trying to hack your data looking for small enough p-values). Often this is done with the best of intentions, as seeing data from an

…

sample size in a replication study to be able to detect a smaller effect size Specifying statistical tests to run ahead of time to avoid p-hacking Nevertheless, as a result of the replication crisis and the reasons that underlie it, you should be skeptical of any isolated study, especially when you

…

perverse incentives, 50–51, 54 Peter, Laurence, 256 Peter principle, 256, 257 Peterson, Tom, 108–9 Petrified Forest National Park, 217–18 Pew Research, 53 p-hacking, 169, 172 phishing, 97 phones, 116–17, 290 photography, 302–3, 308–10 physics, x, 114, 194, 293 quantum, 200–201 pick your battles, 238

…

base rate fallacy in, 157, 158, 170 Bayesian, 157–60 confidence intervals in, 154–56, 159 confidence level in, 154, 155, 161 frequentist, 158–60 p-hacking in, 169, 172 p-values in, 164, 165, 167–69, 172 standard deviation in, 149, 150–51, 154 standard error in, 154 statistical significance, 164