Aditya polumetla in partial fulfillment of the requirements for the degree of master of science
|
|  Yüklə 1.32 Mb.
|
Chapter 4 Experiments and Results This chapter presents the experiments we performed to predict various weather variables using ML methods. We first discuss our experiments to predict the variables temperature, precipitation type, and visibility at various RWIS sites using different machine learning (ML) algorithms. In the next section we discuss our experiments to predict the variables temperature and precipitation type at various RWIS sites using HMMs. For each experiment we present the methodology, the results obtained, and discuss the accuracy of that method in predicting values reported by RWIS sensors and the use of this methodology in detecting RWIS sensor malfunctions. 4.1 Using ML Algorithms to Predict Weather Variables To detect an RWIS sensor malfunction, we will be predicting the weather condition reported by the sensor and comparing it with the value reported by the sensor. As discussed in Chapter 3, we will be using, as input attributes in the dataset, the related information gathered from nearby RWIS sites that belong to the same group and the respective AWOS sites associated with them. We evaluated the performance of each algorithm on the data using the 10-fold cross-validation technique. WEKA, described in Section 2.2, contains implementations of many ML algorithms. It allows the user to select an ML algorithm and apply it to a given dataset. Using the selected algorithm, it builds a model based on the training set provided and uses the model to classify instances in the training set and gives the performance results. A brief description of WEKA along with the command-line statements used to run different algorithms is given in Appendix B. We performed the experiments involving ML algorithms mentioned previously using WEKA. We averaged the results of ten 10-fold cross-validation runs for our ML experiments. 4.1.1 Predicting Temperature In this section we describe the experiments performed, using both classification and regression algorithms, to predict the temperature value at an RWIS site. The experiments were performed using default options provided by WEKA for the respective algorithms. All our datasets were built from the year 2003 data because at the time the experiments were performed 2003 was the latest year for which we had an entire year's data for the selected 13 RWIS sites. 4.1.1.1 Experiment 1: Predicting temperature using regression methods In this experiment we predict the current hour temperature value at an RWIS site using regression algorithms. The temperature data from the RWIS site and the RWIS - AWOS in its group are used to form the feature vector. The results obtained from predicting temperature values were analyzed. Methodology We used the regression algorithms Linear Regression (LR), LeastMedSquare (LMS), M5P, Multilayer Perceptron (MLP), RBF Network (RBF), and Conjunctive Rule (CR) to predict the current temperature at an RWIS site. Our feature vector for the dataset consists of the current and the previous three hour temperature values obtained from the RWIS and the AWOS sites in a group of related sensors (the grouping of sites into sets is described in Section 3.1), and the current hour temperature at the RWIS sites. We calculate the temperature as the difference between the temperature values reported at an RWIS site and the projected hourly temperature (described in Section 3.2.1) at its corresponding AWOS site. For example, if the temperature at the RWIS site 67 is 32°F at time t and the projected hourly temperature at the time t for its corresponding AWOS site, KLYU, is 30°F, then the temperature of site 19 for hour t in the feature vector is 2°F. During the calculation of hourly projected values we used historical averages (from years 1997 to 2004) for a month and the average temperature for a day to extract temperature values as deviations from the average. Using historical data provides additional information apart from what we already have (i.e., the RWIS and the AWOS data). We used the weather data collected from the year 2003 to build our dataset using the described feature vector. For each RWIS-AWOS group we get a different dataset. To predict temperature at an RWIS site we use the dataset describing the sites in that group. The current temperature value at the RWIS site to be predicted is the output attribute and all other attributes in the feature vector form the input attributes. We used the regression algorithms mentioned above and the dataset generated to try to predict the current hour temperature value at the selected 13 RWIS sites. Results Our testing involved ten 10-fold cross-validation runs so each instance in the dataset is predicted 10 times (i.e., one in each cross-validation), the average of these 10 values gives the final predicted value. We used absolute error between the actual value reported by the sensor and the value predicted by the algorithm, to evaluate the performance of the algorithm. Figure 4.1 shows the mean of absolute error and the standard deviation obtained across all RWIS sites for each algorithm. The ordering of algorithms in the figure is according to the order in which algorithms were describing in Chapter 2. The mean absolute error obtained for each individual RWIS site for an algorithm, in a detailed form, are shown in Appendix C.  Figure 4.1: The mean of the absolute error and the standard deviation obtained from predicting temperature across all 13 RWIS sites using regression algorithms. The four methods LR, LMS, M5P and MLP clearly outperform RBF and CR.
Discussion By comparing the mean absolute errors for CR and RBF with other methods, we see that they fail to predict temperature values for all sites. It may be that these algorithms could have performed better if we had significantly tuned the parameters for these algorithms. Mean error for LMS, LR and M5P is seen to slightly less than 1°F and a slightly more than 1°F for MLP. It can be observed from these values that the predictions of temperature value by these algorithms, LMS, LR, M5P and MLP is accurate to ±1°F. The standard deviation value for the mean absolute errors across different sites for an algorithm measures the variation in the error values across the sites. A small standard deviation suggests the model is fairly independent of the site location and can predict with the same accuracy on any site. This is favorable as one model can be used across all sites rather than having a separate model for each individual site. The CR and RBF algorithms show higher standard deviations when computed with the rest of the algorithms used, but this is likely a function of their high error. The best standard deviation results are obtained from M5P with 0.058 as the standard deviation value across sites. Combining the results and giving priority to the algorithms that has a low absolute error and has a similar behavior across sites, we find that M5P performed the best followed by LR. It can be clearly seen that CR and RBF did not do well concerning prediction of temperature and were not used in the the other experiments that followed. To detect RWIS temperature sensor malfunctions models built from M5P or LMS can be used. When the difference between the error reported for an hour and the mean absolute error obtained from testing the model is greater than 1.96 standard deviations, with standard deviation of error calculated from the test results, we can say with 95% accuracy that the sensor has failed. 4.1.1.2 Experiment 2: Predicting temperature using regression methods, with precipitation type included as inputs. In this experiment we predict the current hour temperature value at an RWIS site using regression algorithms. The temperature data from the RWIS site and the RWIS - AWOS sites in its group along with the precipitation type seen at the RWIS sites are used to form the feature vector. The results obtained from predicting temperature values were analyzed. Methodology Due to the fact that presence of precipitation affects the temperature at a location, we added it to the feature vector, so that the model built uses this information in predicting temperature. We used the regression algorithms LMS, LR, and M5P to predict the current temperature class value at an RWIS site. We use these three algorithms as they were the top three of the algorithms tried in Experiment 1 (see Section 4.1.1.1) and are arranged in ascending order with respect to the mean absolute errors across all sites (see Figure 4.1). Our feature vector for the dataset consists of the current and the previous three hour temperature values from the RWIS and the AWOS sites from a group of related sensors and the precipitation type observed at the current hour at the RWIS sites. We used the weather data collected from the year 2003 to build our dataset using the described feature vector. The current temperature at the RWIS site to be predicted is the output attribute and all other attributes in the feature vector form the input attributes. We use the regression algorithms mentioned above and the dataset generated to try predict current hour temperature class value at the selected 13 RWIS sites. Results We use the mean absolute error obtained from the ten 10-fold cross-validation runs to evaluate the performance of the algorithm. Figure 4.2 shows a comparison of the mean absolute error and the standard deviation averaged for all RWIS sites, using the regression algorithms obtained from this experiment and from Experiment 1 (see Section 4.1.1.1) in which precipitation type information was not included. The mean absolute error obtained for each individual RWIS site for an algorithm, in a detailed form, from this experiment are shown in Appendix C. Discussion Of the three algorithms (LR, LMS and M5P) used, we see that M5P shows better results in predicting temperature with lesser mean absolute error and consistency in predictions across the sites (with respect to the standard deviation of errors across various sites). Significant variation of absolute errors reported by these three algorithms was not seen, with all them predicting temperature with an accuracy close to 0.95°F.  Figure 4.2: The comparison of mean of absolute error and standard deviation obtained from predicting temperature from Experiment 1 without precipitation (Section 4.1.1.1) and Experiment 2 (Sections 4.1.1.2) with precipitation.
We also note that including precipitation type in the dataset as an additional source of information does not decrease the error of the model built. It can be observed from Figure 4.2 that the mean absolute error for the algorithms LR, LMS and M5P is higher for this experiment when compared with Experiment 1. It was observed the mean of absolute error increased by approximately 0.09°F when precipitation type was included in the dataset. 4.1.1.3 Experiment 3: Predicting temperature class using classification methods In this experiment we predicted the current hour temperature class value at an RWIS site using classification algorithms. We discretized the temperature values for its use in classification algorithms. The temperature data from the RWIS site and the RWIS-AWOS sites in its group along with the temperature's class value at the RWIS sites are used to form the feature vector. The results obtained from predicting temperature class values were analyzed. Methodology We used the classification algorithms J48 decision trees and Naive Bayes (NB) to predict the current temperature class value at an RWIS site. Our feature vector for the dataset consists of the current and the previous three hour temperature values from the RWIS and the AWOS sites from a group of related sensors and the class value for the current hour temperature at the RWIS sites. We discretized the temperature value using the method described in Section 3.2.2 and the class distribution was set according to the example mentioned at the end of that section which divides temperature into nine different classes. We used the weather data collected from the year 2003 to build our dataset using the described feature vector. The class value for the current temperature at the RWIS site to be predicted is the output attribute and all other attributes in the feature vector form the input attributes. We used the classification algorithms mentioned above and the dataset generated to try predict the class for the current hour temperature at the selected 13 RWIS sites. Results We used the absolute distance between the class value of the temperature reported by the RWIS sensor and the temperature class predicted to evaluate the performance of the classification algorithms used. A distance of 0 indicates that the predicted class value is same as the class value reported. The percentage of instances that were reported with a distance ranging from 1 to 6, for both J48 and NB used for predictions, are shown in the Figure 4.3. F  igure 4.3: The distance between actual and predicted temperature class obtained from J48 and Naive Bayes algorithms. Discussion We see that from the results in Figure 4.3 that the J48 outperforms the the Naive Bayes algorithm by classifying 93.6% of the instances in the dataset whereas only 32.337% instances were correctly classified by the NB algorithm. No instances were reported having a distance of more than 3 between actual and predicted class value when prediction was done using J48. As 99.435% of the instances in the dataset were predicted with a distance of either 0 or 1, using the J48 algorithm, we can predict RWIS sensor malfunctions when the distance between the actual and predicted temperature class value is greater than 1. For all the malfunction cases in our dataset the difference in the class value is 2 or greater, thus we can say that sensor malfunctions can be detected with high accuracy with the J48 algorithm. 4.1.2 Predicting Precipitation Type This section describes the experiments we performed using the the classification algorithms to predict precipitation type at an RWIS site. The experiments were performed using the default options provided by WEKA for the respective algorithms. We employ only classification algorithms because precipitation type is reported by RWIS sensors in the form of class values and classification algorithms are used to classify the given instance into class values taken by the output attribute. The dataset was built from the year 2003 data because at the time the experiments were performed 2003 was the latest year for which we had an entire year's data for the selected 13 RWIS sites. 4.1.2.1 Experiment 4: Predicting precipitation type using classification methods In this experiment we predict the current hour precipitation type reported at an RWIS site using classification algorithms. The temperature data from the RWIS site and the RWIS- AWOS sites in its group along with the precipitation type reported at the RWIS sites are used to form the feature vector. The results obtained from predicting precipitation type were analyzed. Methodology We used the classification algorithms J48 decision trees, Naive Bayes (NB) and Bayesian Networks (Bayes Nets) to predict the precipitation type for the current hour at an RWIS site. The K2 algorithm is used to learn the Bayesian network structure. Our feature vector for the dataset consists of the current and previous three hour temperature values from the RWIS and the AWOS sites in a group of related sensors and the precipitation type observed at the current hour for the RWIS sites. We included the temperature information to help in the prediction process as there is a correlation between temperature and precipitation observed at a location. Precipitation information from AWOS sites was not used because the effect of precipitation is localized and does not effect the occurrence of precipitation at nearby and other locations.  Figure 4.4: The classification error and the standard deviation obtained from predicting precipitation type across all 13 RWIS sites using classification algorithms.
We used the weather data collected from the year 2003 to build our dataset using the described feature vector. The precipitation type at the current hour for the RWIS site to be predicted is the output attribute and all other attributes in the feature vector formed the input attributes. We used the classification algorithms mentioned above and the dataset generated to try predict current precipitation type for the selected 13 RWIS sites. Results When classification algorithms are run using WEKA, the output is presented as a confusion matrix and statistical results such as the classification error, root mean squared errors and the percentage of correctly classified instances are given by WEKA. Figure 4.4 shows the classification error (as reported by WEKA) and standard deviation obtained from predicting precipitation type across all RWIS sites for an algorithm. These values were obtained after averaging the reported values for each cross-validation run. The classification error and standard deviation values obtained for each individual sites for the  Figure 4.5: The percentage of instances with precipitation present and with no precipitation present predicted using classification algorithms.
algorithms used in a detailed form are shown in Appendix C. For predicting precipitation type, we use the percentage of instances that were correctly classified and incorrectly classified when precipitation was present and when no precipitation was present to determine the accuracy of the algorithm used. Figure 4.5 shows the percentage of instances that were reported as precipitation present and no precipitation present in the actual data. For each algorithm the figure shows the percentage of instances these values were predicted correctly and incorrectly when precipitation was present and when precipitation was not present. A detailed analysis of the these percentage values of correctly and incorrectly classified instances for the algorithms used across each individual site is given in Appendix C. Discussion From the classification errors reported by WEKA for the three regression algorithms used, we see that J48 performs better in predicting precipitation type but a high standard deviation across sites shows that it is not consistent in predictions across different sites, with site 35 having 0.077 as classification error and site 67 with 0.356 as its classification error. Other than site 67 none of the sites have classification error above 0.26 which is lower than the mean absolute errors reported by NB and Bayes Net. Some RWIS sites report precipitation as present or not present while the remaining sites report the type of precipitation observed. In order to compare sites, we combined all different type of precipitation together and reported precipitation as present. This allows comparison o the percentage of instances when precipitation was correctly classified between sites. Predicting precipitation is a challenging task. It observed from analyzing the RWIS data that the instances with precipitation present were very few, this is because precipitation does not continue for a long time and may be present for an hour or two leading to few observations reported over a period of time. As seen in Figure 4.5, Naive Bayes and Bayesian Networks predict only about 19% of the instances as no precipitation present when in the actual data 81% of the instances were reported as no precipitation. They perform poorly for classifying no precipitation. We found that J48 classifies instances with no precipitation present correctly, with identifying 75% of the instances as no precipitation present when in actual data 81% of the instances had reported no precipitation. But it fails to report the presence of precipitation with the accuracy it predicts precipitation. The detection of RWIS sensor malfunctions using prediction of precipitation type is a tough task because of the failure of the algorithms to correctly classify precipitation. We found that combination of J48 and Bayes Net can be used to detect malfunctions, with absence of precipitation being calculated through J48 and Bayesian Networks being used to report presence of precipitation. We choose Bayesian Networks over Naive Bayes because of its smaller mean absolute error and standard deviation. Due to varying accuracies prediction among different sites (see Appendix C), each individual site requires its own model and specific percentages with which they correctly classify precipitation to identify the sensor malfunctions. For example, for site 62, J48 mis-classifies 3.84% of instances when predicting no precipitation and Bayes Nets mis-classifies 0.77% of instances when reporting presence of precipitation. For this site we can say that when J48 predicts incorrectly when no precipitation is present and with the difference in classification error for the site (i.e., 0.062 for site 62) and the absolute error reported by WEKA is greater than 1.96 standard deviations (standard deviation calculated from the errors obtained on the test set) then we can say with 95% accuracy that the sensor has failed. The same approach is followed when Bayes Nets wrongly reports absence of precipitation. Combination of J48 and Bayes Nets produce high accuracy in detection sensor malfunctions when each of them is individually responsible for classifying absence of precipitation and presence of precipitation respectively. 4.1.3 Predicting Visibility This section describes the experiments we performed using regression algorithms to predict visibility at an RWIS site. The experiments were performed using the default options provided by WEKA for the respective algorithms. 4.1.3.1 Experiment 5: Predicting visibility using regression methods In this experiment we predict the current hour visibility reported at an RWIS site using regression algorithms. The temperature data from the RWIS site and the RWIS-AWOS sites in its group along with the precipitation type and visibility reported at the RWIS sites are used to form the feature vector. The results obtained from predicting visibility were analyzed. Methodology We used the regression algorithms LR, LMS, M5P, and MLP to predict the visibility for the current hour at an RWIS site. As visibility is affected by the presence of precipitation, we included precipitation as an attribute in the feature vector. Our feature vector for the dataset consists of the current and previous three hour temperature values obtained from the RWIS and the AWOS sites in a group of related sensors, the precipitation type observed at the current hour for the RWIS sites and the visibility at the current hour the RWIS sites. The RWIS sites that report visibility are included in the feature vector. We used weather data collected from the years 2002 and 2003 to build our dataset using the described feature vector. The visibility at the current hour at the RWIS site to be predicted is the output attribute and all other attributes in the feature vector form the input attributes. We used the regression algorithms mentioned above and the dataset generated to try predict current visibility for the RWIS sites that report it. Results The mean absolute error obtained from site 67 in RWIS Set 1 was excluded when averaging values for the sites used in prediction, this was done because site 67 reports visibility up to 10 miles whereas all other sites report only up to 1.09 miles. Figure 4.6 shows the mean of absolute error and standard deviation obtained from predicting visibility across all RWIS sites, excluding site 67, for an algorithm. The overall mean absolute errors were obtained after averaging the values reported for each cross-validation run. The mean absolute errors obtained for each individual sites with respect to the algorithm used are given in Appendix C. |
Figure 4.6: The mean of the absolute error obtained from predicting visibility across the RWIS sites that report visibility using various algorithms.