BERNARD BARUCH COLLEGE
MTH 4130 MATHEMATICS OF DATA ANALYSIS (FALL 2020)
Final
The aim of this take-home final is to illustrate the techniques discussed during the first lecture to predict
temperature as a function of time. The data to be used are historical records of temperatures across the United
States. Data are available on the Noaa website1
and we will focus on the Hourly02 directory. The time is
specified by UTC DATE, UTC TIME, LST DATE and the temperature by T HR AVG.
Part one. As you can see data refer to multiple weather stations across the United States. The data set is not
consistent, for some of the stations there are recordings going back to the year 2000 while for some other weather
stations data recording starts later in time. In addition, it may happen that some of the data are missing only
for few days of the year or few hours of a given day. We will focus on studying temperature relative to the
weather station in Asheville (North Carolina) as a function of time.
(1) Plot the data over different time intervals (e.g. one day, one week, and one month) and realize that
there are periodic variations happening at different time scales. Temperatures variations of night and
day cycles happen with a period of 24 hours while there are seasonality variations happening on the
time scale of months.
(2) Identify a good set of features (all of them have to be a function of time at this point of the project)
to be used with linear regression to describe the time series. Clearly, good candidates for this task are
periodic sine and cosine functions of time with periods that take into account the variations observed
at the previous point. Specify wich features you are using.
(3) By using linear regrassion, build a first predictive model that is valid only for the month of May. In
order to do that you can isolate the month of May from each year of the time series and use hourly
temperature relative to that month only to predict temperatures in May.
Blind cross validation does not really make sense in the context of time series. A better idea is to
train your model using all the data except the one referring to the last year available (that should be
2020) on which you can test your model. Focusing on the prediction of a given month should make
features selection a bit easier since seasonal variations of the temperature should not play such a big role.
The more features you use the more the complexity of your model increases. Show the phenomenon of
overfitting when too many features are included in the model.
(4) Try to build a model that works for the entire year. Now you should include features (again functions
of time) that take into account seasonal variation of the temperature. Include all the data up to 3
days before the day/time you want to predict. So if you are predicting the temperature on May 5th at
4pm. Use all the temperature available up to May 2 at 4pm. Compare the performances of these two
models (the one using only data from May and the one using data over all the year) to predict hourly
temperatures during the month of May.
(5) Build another model using KNN. Here overfitting happens as you decrease the number of points K used
to compute the local averages used in KNN. Explain how you choose K in this context.
(6) How do the linear regression model and KNN compare in terms of predictions?
Part two. Include in your model the features as T MAX and T MIN and P CALC. Do they help to predict the
temperature?
In addition, in this part you should analyze your model from the statistical point of view. How many
coefficient are significantly different from zero in your linear regression model? How big is their variance?
Part three - optional for extra credits. By looking at the data you may realize that there are temperatures
variations that are not regular in time. These are weather systems passing over the weather station. They are
hard to predict and, for instance, cannot be captured with the linear regression model used at the previous
point. Come up with a strategy to predict these events for a given weather stations. This strategy should be
based on the information that can be captured from nearby weather stations by considering that a weather
system over a given region it is likely to move to nearby regions with a pattern that most of the time is regular
(from west to east).
In this case it is useful to see if there is a correlation between signals of different stations. The correlation
analysis should be done taking into account times lag of the signals due to the fact that it takes time for a
weather system to move from a region to another.
1https://www.ncdc.noaa.gov/crn/qcdatasets.html