ATHK1001 ANALYTIC THINKING: ASSIGNMENT 1, 2023
Due date: 11:59pm Friday, March 31st (Week 6). Late penalty of 5% per calendar day applies.
Online submission: All submissions are to be made online on the ATHK1001 Canvas website.
Submissions will be checked for plagiarism. Artificial Intelligence tools such as ChatGPT that assist
writing are not permitted.
Incorrect submissions: If you discover before the closing date that the file you submitted on
Turnitin was incorrect, and let us know, you may be given the option to resubmit a corrected
version with a 50% penalty or the relevant lateness penalty, whichever is greater.
Word length: 750 words across all questions (excluding references in Question 13). A penalty of
10% will apply to papers that exceed this limit by more than 10%, a 20% penalty if you exceed
20% of the limit, and 30% if you exceed the limit by 30%.
Total marks: 60 (15% of total grade for class)
Background and Aims
A useful skill when dealing with data is being able to estimate plausible answers to questions you
don’t know the answer to. A strategy for doing this is embodied by what are known as “Fermi problems.”
These are numerical estimation problems that break down a difficult estimation problem into steps. A
common example is “How many piano tuners are there in Chicago?” Most people have no idea of what
the right answer may be, but it can be broken down into a set of estimations about which people have
more confidence: How many people are there in Chicago? How many families? What proportion of
families have a piano? How often does a piano have to be tuned? How long does it take to tune a piano?
How many days a year would a piano tuner work? People can come up with reasonable estimates for the
sub-questions and then combine them to make a reasonable estimate for the main questions. They are
called Fermi problems after the Nobel-prize winning physicist Enrico Fermi who reportedly estimated the
force of the first atomic explosion from how far a dropped piece of paper travelled. He was famous for
being able to use such meagre pieces of information to derive surprisingly good answers to questions.
Most famously Fermi problems are the basis for the Drake equation for estimating the number of extra-
terrestrial civilizations. In both science and engineering education Fermi problems are used to show
students the power of deductive thinking, introduce mathematical modelling, and prepare them for
experimental laboratory work. As such they have been used at educational levels ranging from primary to
tertiary (see ?rleb?ck & Albarracín, 2019).
?rleb?ck and Albarracín (2019) conducted a systematic literature review and identified 91 articles
that addressed Fermi problems. Forty-three of these articles are described as empirical studies focused on
teaching or learning using Fermi problems. These have demonstrated that Fermi methods can be taught,
but ?rleb?ck and Albarracín report that although many of the articles advocate for the benefits of
teaching Fermi problems to students none of the reviewed research focused on explicitly establishing
evidence supporting the argument that Fermi problems improve students’ estimation skills. So there
appears to be no published research that addresses the basic question: Does treating a problem as a Fermi
problem lead to better estimates?
The experiment conducted for this assignment addressed the question of whether participants
make more accurate estimates for Fermi problems rather than non-Fermi problems. It does by treating the
same question as a Fermi problem when it is broken down into sub-questions, or a non-Fermi problems
the target question is presented alone. To evaluate this issue, we tested a set of hypotheses using the data
we collected in tutorials in Week 2.
Method
Participants
A total of 197 students from analytic thinking course (ATHK1001) participated as part of a class
experiment were analysed. A larger number participated but were eliminated from the analysis due to
2
their data not being correctly recorded, they completed too little of the experiment, or they did not give
consent to having their data analysed. Of the analysed participants 92 were female, 100 were male, and
they had a mean age 19.3 years.
Materials
All question used were given as either Non-Fermi problems or Fermi problems. Fermi problems
asked the participant to answer a series of four or five sub-questions before answering the main question.
For example, the Fermi problem version of the question “How many piano tuners do you think there are
in Chicago?” first asked:
What do you think is the population of Chicago?
How many pianos do you think there are in Chicago?
How many hours do you think it takes to tune a piano?
How often do you think pianos are tuned over a ten-year period?
How many hours a year do you think the average piano tuner works?
The Non-Fermi problem version just asked “How many piano tuners do you think there are in Chicago?”
Two sets of nine questions each were used. Most Fermi problems presented in lists of Fermi
problems seems to encourage estimation by multiplication sub-question answers, such as appears to be
the case for the Chicago piano tuners question. However, some questions seem to encourage addition of
sub-question answers. For example, the question “What is the total military budgets of the members of
the UN Security council (USA, Russia, China, UK, France), in millions of $US?” was broken down into
the sub-questions:
What is the military budget of the USA, in millions of $US?
What is the military budget of Russia, in millions of $US?
What is the military budget of China, in millions of $US?
What is the military budget of the UK, in millions of $US?
What is the military budget of France, in millions of $US?
So, we constructed a set of nine multiplicative questions and a set of nine additive questions. A list of the
nine questions for each set together with their sub-questions can be found in the Appendix. Every
participant received both sets of questions: one set as Fermi problems (i.e., with sub-questions) and the
other set as Non-Fermi problems. The experiment had two conditions: in the “Multiplicative condition”
participants answered the nine multiplicative questions as Fermi problems and the nine additive questions
as Non-Fermi problems, where as in the “Additive condition” participants answered the nine additive
questions as Fermi problems and the nine multiplicative questions as Non-Fermi problems.
Procedure
During tutorials for the class Analytic Thinking at the University of Sydney participants
completed the experiment individually on computers in the classroom or online. All participants were
randomly assigned by the computer to either “Multiplicative condition” or the “Additive condition” and
then they answered either their nine Fermi problems then their nine Non-Fermi problems, their nine Non-
Fermi problems then their nine Fermi problems. The order in which they answered the problems was
varied randomly. Responses to each question or sub-questions were entered into textboxes on a webpage,
and participants had to enter numbers in response to every question before they cold advance to the next
page.
After completing the experiment participants indicated whether or not they consented to having
their data included in the data set.
Hypotheses
We proposed four hypotheses to investigate how participants performed on Fermi and Non-Fermi
problems.
3
First, we will test whether Fermi and Non-Fermi problems differed in terms of accuracy, and based on
the wide endorsement of Fermi problems we predicted that participants would do better on Fermi
problems.
Hypothesis 1: Mean accuracy for Fermi problems will be different than for Non-Fermi problems.
We predict accuracy will be greater for Fermi problems.
Hypotheses 2 and 3 tested if which set of questions were answered as Fermi problems would make a
difference. We had no reason to predict a difference.
Hypothesis 2: Mean accuracy for Fermi problems will not be differ between “Multiplicative
condition” and the “Additive condition”.
Hypothesis 3: Mean accuracy for Non-Fermi problems will not be differ between “Multiplicative
condition” and the “Additive condition”.
Hypothesis tested whether how accurate participants were on Fermi problems was associated with their
accuracy on Non-Fermi problems.
Hypothesis 4: Accuracy for Fermi problems will correlate with accuracy for Non-Fermi problems.
Results
The data set for our class can be found on the Canvas site for ATHK1001 under “Assignment 1”.
This assignment description can be found there as well as an Excel file called “Assignment1_dataset.xls”.
This Excel file contains all the data for the assignment and has 197 data lines, one for each participant.
Each participant has values for four variables, and the values of each variable are in a single column of
the file.
The first variable is an arbitrary id number generated by the computer. The “condition” column
gives each participant’s condition: “1” means participants were in the “Multiplicative condition” and “2”
means they were in the “Additive condition”. The third variable is the participants accuracy on their
Fermi problems. The third variable is the participants accuracy on their Non-Fermi problems.
Accuracy was calculated based on whether participants’ answers were the same order of
magnitude as the correct answer. The order of magnitude of an answer is an approximation of the
logarithm (base 10) of a value, and can be understood as the numbers of digits in a integer answer minus
1. For, example, any answer between 100 and 999 would have and order of magnitude of 2, because 100
is 102. In estimation which is expected to be inaccurate often the aim is to estimate to the correct order of
magnitude, and sometimes this is explicitly seen as the goal when setting a question up as a Fermi
problems. The Appendix presents the correct order of magnitude for each question.
For a participant’s Fermi problems how many of their answers had the correct order of magnitude
was countered up and then divided by the number of questions they answered, so the Fermi accuracy was
a proportion between 0.0 and 1.0. Similarly, participants’ Non-Fermi accuracy was the proportion of their
Non-Fermi problems that their answers had the correct order of magnitude.
WHAT YOU WILL WRITE
Your task is to analyse the data in order to test the four hypotheses proposed above. You will do this by
addressing each of the following 13 questions. Answer all questions with complete sentences, not with
just numbers, notes or tables. You will be penalized if you do not use complete sentences. Do not include
the text of the questions in your assignment (this will trigger a plagiarism warning), but you should
include the number of the question being addressed.
A note on how many digits to report in your answers. Excel will give answers with a huge string of
digits, many more than are necessary to understand the results, so the question arises of how many digits
to report in your answers? The convention is to use the same number of significant digits as the data used
4
in the calculation. Significant digits are the digits excluding the leading and trailing zeros. Therefore
1.23, 0.0000123 and 12300000000, all have three significant digits.
1) For Fermi accuracy and Non-Fermi accuracy report the means and standard deviations. (4 marks)
2) Based on the means you calculated in Question 1 use a t-test to test Hypothesis 1, that mean accuracy
for Fermi problems will be different than for Non-Fermi problems. Report the p-value for the t-test and
state clearly whether or not Hypothesis 1 was supported, and state why. (Note that we will be discussing
hypothesis testing in lectures in Week 4 and practicing using Excel to test hypotheses in tutorials in Week
5. So, you may need to wait to answer this question until we have covered the relevant material in class.)
(3 marks)
3) For Fermi accuracy report the means and standard deviations for both the “Multiplicative condition”
and the “Additive condition”. (4 marks)
4) Based on the means you calculated in Question 3 use a t-test to test Hypothesis 2, that mean accuracy
for Fermi problems will not be differ between “Multiplicative condition” and the “Additive condition”.
Report the p-value for the t-test and state clearly whether or not Hypothesis 2 was supported, and state
why. (3 marks)
5) For Non-Fermi accuracy report the means and standard deviations for both the “Multiplicative
condition” and the “Additive condition”. (4 marks)
6) Based on the means you calculated in Question 5 use a t-test to test Hypothesis 3, that mean accuracy
for Non-Fermi problems will not be differ between “Multiplicative condition” and the “Additive
condition”. Report the p-value for the t-test and state clearly whether or not Hypothesis 3 was supported,
and state why. (3 marks)
7) Present one graph that shows the means for absolute accuracy for Fermi accuracy and Non- Fermi
accuracy for the “Multiplicative condition” and the “Additive condition”. To do this you will need to
present a total of four means, and for each of these means include error bars (error bars visually indicate
standard errors above and below the mean). Either a bar or line graph could be used for this, but a bar
graph is preferable. (6 marks)
8) Consider your answers to Questions 4 and 6. What do they suggest regarding whether the type of
Fermi matters and if it matters how it matters? Explain why you drew this conclusion by referring to our
hypotheses and how strongly you draw it. (4 marks)
9) Calculate the correlation between a participant’s accuracy Fermi accuracy and Non-Fermi accuracy.
Test Hypothesis 4 and state whether Hypothesis 4 was supported, and why. What are the implications of
your tests of this hypothesis? (5 marks)
10) Identify three different issues with the way we collected data which could limit our ability to draw
conclusions from it. These issues could relate to one or more of the hypotheses. Clearly differentiate the
three issues as “Issue 1”, “Issue 2” and “Issue 3” and explain how each of these issues relates to a data
collection consideration raised in ATHK1001 lectures, readings or tutorials. For each issue suggest a way
it might be resolved in future research on this topic or if it cannot be resolved then explain why. (12
marks)
11) Based on your analyses presented for previous questions, state whether we have found evidence that
presenting questions as Fermi problems improves participants’ accuracy. Explain why you drew this
conclusion by referring to our hypotheses and how strongly you draw it. (4 marks)
5
12) Summarize what you think the data analysis you have carried out for this assignment tells us about
Fermi problems. Do they add to ?rleb?ck and Albarracín (2019) review of the state of research into
Fermi problems, and if so state specifically how they do so. Explain your responses with explicit
reference to the results of your testing of the hypotheses and possibly the issues you raised in Question
10. (6 marks)
13) Include a reference section which lists the full reference for any paper you have cited when
addressing these questions. You must cite ?rleb?ck and Albarracín (2019) in Question 12, and include
citations where ever appropriate. You should use APA style for citations and references, but we will
accept other standard journal article referencing formats. (2 marks)
Reference
The use and potential of Fermi problems in the STEM disciplines to support the development of
twenty?first century competencies.
Authors: J. B. ?rleb?ck and L. Albarracín
Source: ZDM, volume 51, pages 979–990.
THIS IS NOT IN A STANDARD REFERENCING FORMAT; YOU WILL NEED TO REFORMAT
THIS FOR QUESTION 13 USING A STANDARD FORMAT (E,G., APA STYLE)
Note that this paper provides background so you do not have to understand every aspect of it. Also, you
do not have to use other sources for answering these questions, but if you do then you must correctly cite
and reference these sources.
Formatting Recommendations
Our preferences
Use the font “Times New Roman”, 12-point size, and double-space all the lines.
Indent the beginning of each paragraph using one tab space.
Use APA referencing style