Aditya polumetla in partial fulfillment of the requirements for the degree of master of science
|
|  Yüklə 1.32 Mb.
|
Discussion We found that the LMS and the M5P algorithms predict visibility with almost the same absolute error, with LMS being a little bit higher. But the standard deviation of errors across various sites was low for M5P when compared to LMS. Thus, we can say M5P can be used to report visibility across different sites accurately and its predictions are site independent and a model created using one site can be used to predict values at another site. We can use M5P to detect sensor malfunctions. We can say that when the the difference between error from prediction and the mean absolute error for M5P is greater than 1.96 standard deviations, with standard deviation of calculated from the error values obtained during cross-validations, the sensor is malfunctioning. 4.2 Using HMMs to Predict Weather Variables Based on the instance given, an HMM tries to predict the most probable path taken across the states. For the weather data the “path” is the predicted values of the weather sensor using the surrounding sensors as additional information. Both HMMs and classification algorithms can be used to classify a discrete attributes such as precipitation type. When the data is present in the form of time series, such as hourly precipitation type observations for a time will form a hourly series of values, HMMs can be used to identity the path of predicted values from which we can further obtain the values observed through the path. We used the modified Viterbi algorithm (see Table 3.2) to predict the symbol observed when a state in the most probable path is reached for the given instance. HMMs require a training dataset to build the model, that is to estimate the initial state, transition and emission probabilities. The test dataset is used to evaluate the model. Multiple n-fold cross-validation can be used to estimate the model performance. As HMMs require the presence of all symbols in the symbol sequence, any day with no value (i.e., missing) reported by a sensor at an hour for any of the RWIS sites in the set, was omitted from the dataset used for predictions. HMMs require class value information, thus we discretized the temperature for using it with HMMs by following the method described in Section 3.2.2. We found that with larger class distribution for sites to be used for prediction helps in better comparison between the predicted and actual class values whereas the nearby sites are used to provide additional information about the surrounds weather conditions and they need not be taken with greater precision. Thus for the RWIS site that were used for predictions, the temperature was broken down into nine classes based on the value obtained for number of standard deviations the actual temperature value differs from the predicted value, the classes were divided as
For the RWIS sites that were appended along with the site to be predicted in the class string, the temperature was broken down into five classes with the classes divided as
For example, to predict temperature values at site 19 we use temperature values from site 27 and 67 as additional information. The temperature of site 19 was be broken into 9 classes and temperature at sites 27 and 67 will be broken into 5 classes. In the following experiments related to HMMs, we trained the HMM (Baum-Welch algorithm) using the methods available in the 'HMM Toolbox for MATLAB' developed by Murphy [1998] and the modified Viterbi algorithm was used for testing. We performed ten iterations of the Baum-Welch algorithm during the training of the HMM. We set the number of states in the model to 24 and each state was allowed to emit all possible symbols obtained from the training set. As we are using hourly readings for a day, each instance in the dataset is a string of symbols with a length of 24. 4.2.1 Predicting Temperature This section describes the experiments we performed using HMMs to predict the class value of the temperature at an RWIS site. 4.2.1.1 Experiment 6: Comparison of two methods for training HMM Two different methods were devised to train the HMMs. In this experiment we predict the temperature class value reported at an RWIS site using HMMs. The temperature class information from the RWIS site and the other RWIS sites in its group are used to form the feature symbols. The results obtained from predicting temperature class value were used to compare the two methods. Methodology We trained the HMM to predict the temperature class value in two different ways Method A: In this approach we merge all the data from a group and then train the HMM using this data. As noted earlier, the site being used for prediction has the temperature split into 9 classes and other sites in the group have temperature split into 5 classes. Thus, for each site we have its own merged dataset. We create the dataset in which each instance is a string of symbols representing hourly readings per day, where a symbol for an hour was obtained from appending that hour's temperature class value for all of the RWIS sites in a group together. For example to predict temperature class values at site 19 we will be using the temperature classes from sites 27 and 67, if the temperature class values at a time t seen at sites 19, 27 and 67 are 6, 3 and 2 then the class string generated for the time t will be 632. We estimated the initial state, transition, and emission probabilities by applying the Baum-Welch algorithm on this dataset with combined class information. Method B: In this approach the data from the sites used is not merged. Data from each site is used for training the model and the emission probabilities of all sites in the group are multiplied to obtain a single emission matrix that is used in the final model. We created a separate dataset for each RWIS site, in which each instance is a string of symbols representing hourly temperature class value per day. We used the dataset for each RWIS site to estimate the initial state, transition and emission probabilities using the Baum-Welch algorithm. The emission probability matrices obtained for each of the RWIS sites in a group were then joined together by multiplying the respective values in each box in the matrix. The resultant emission matrix was made stochastic, that is, all rows and columns are made to sum to 1. We then applied the m-estimate upon the probability values in the matrix, using m = 20 and p = 1 / #classes, to make the probabilities with very small values bigger, where in our case the number of classes denotes the total number of class values taken by the variable we are trying to predict. By applying the m-estimate we get new values in each block of the matrix given by matrix(row.col) = (matrix(row,col) + mp) / (1 + m) For example, in the matrix shown below we apply m-estimate on it using m=20 and p=½ (considering the number of classes be equal to 2) to get the resultant matrix.
We used the initial state and transition probabilities for the RWIS whose temperature value is to be predicted in the model. At the time of performing these experiments we had data from RWIS sites from 2002 to April 2004, we used the entire data available for building the datasets. We used the temperature values at the RWIS sites for the years 2002 and 2003 in the training set while test set contained temperature information from January 2004 to April 2004. We discretized the temperature values, by breaking the temperature into nine classes. Training was done using the two methods mentioned above and temperature class value was predicted for instances in test data using the modified Viterbi algorithm. Results We used the absolute distance between the actual and predicted class values for temperature as a measure to estimate the performance of the algorithm. Figure 4.7 shows the percentage of instances in the training set with each distance between the actual and predicted class averaged across the results from different RWIS site, when the HMM is Figure 4.7: The percentage of instances with each distance to actual value when the HMM is trained using the two different methods. trained using Method A and Method B. A distance of zero indicates a perfectly classified instance. Discussion Smaller distances between the actual and predicted values indicate more accuracy in predictions. From Figure 4.7 we see that Method A classified the more instances at with distance 1 or less whereas Methods 2 had comparatively lesser number of instances classified with distance below 2. This indicates that Method A outperforms Method B. As Method A gave better results we will be following this approach for training the HMM in the experiments performed in the methodologies described next. 4.2.1.2 Experiment 7: Predicting emperature class using HMMs In this experiment we predict the temperature class value reported at an RWIS site using HMMs. The temperature class information from the RWIS site and the other RWIS sites in its group are used to form the feature symbols. Methodology Using temperature data collected for each RWIS site, ranging from January 2002 to April 2004, we created a dataset for each RWIS site to be used for predictions. In this dataset each instance is a string of symbols representing hourly readings for a day, where a symbol for an hour was obtained from appending that hour's temperature class value for all RWIS sites in a group together. In each class string the RWIS site predicted was taken first and the other sites in the group were added in order of their nautical distance from the concerned RWIS site. This order is useful while finding the closest symbol when a class string seen in the test set is missing from the training set used. We estimated the performance of HMM in predicting the current hour's temperature class value at an RWIS sites using the dataset of the group this site belongs to and applied a single 10-fold cross-validation to estimate the model evaluation. Results We evaluated the performance of the HMM using the absolute distances between the actual and predicted values. The respective distance values for all RWIS sites in a group were averaged so as to reflect on the performance of the HMM in predicting temperature class values for that respective set. Figure 4.8 shows the percentage values for each distance for the three sets, with the percentage calculated as the number of instances classified with a certain distance over the total instances present in the dataset, with each instance being an hourly reading. The percentage for each distance across all the RWIS sites used for prediction is shown in Appendix C. Discussion We do not combine the results of different sets to reach an overall percentage value as the format of the symbol sequence in dataset for each site was different. It can be seen from Figure 4.8 that the most of the temperature class values are predicted with a distance of 1, F  igure 4.8: Percentage of instances having a certain distance from the actual class value when predicting temperature class using HMMs. for all sets. This is also reflected in the results from individual sites shown in Appendix C, with the exception where a distance of 2 is seen most of the times. The percentage of instances predicted with a distance of 3 or less in the groups 1, 2 and 3 are 99.75%, 99.43% and 94.34% respectively. Approximately 3% of the data in Group 3 is predicted with a distance of 4, which is due to the results obtained for RWIS site 67 in which approximately 15% of the data is predicted with distance 4. Taking aside the results for site 67, we can say that, when the distance between the actual temperature class and the predicted class is more than three then there is a malfunction in the RWIS temperature sensor. The erratic behavior of site 67 may be due to presence of errors in it. 4.2.1.3 Experiment 8: Site independent prediction of temperature class using HMMs. In this experiment we predict the temperature class value reported for a site in an RWIS group. A single dataset was generated for a group that has the first class value in the class string as the class value to be predicted. The temperature class information from the RWIS sites in a group are used to form the feature symbols. The results obtained from predicting temperature class were analyzed. Methodology In order to increase the size of data used for predictions and to perform site independent predictions we create a single dataset for each RWIS group. Each of the RWIS sites in a group was taken as the predicted site and the symbol strings obtained from it were appended to the dataset. The class string was constructed using the predicted site's class value which is added first followed by the class values of the other sites in the group with the predicted site are added. The order of the near sites was chosen with respect to the distance from the predicted site, thus the site closest to the predicted site was added first and so on. For example, to create a single dataset for the RWIS Set 1 we first add data instances taking RWIS site 19 as the site to be predicted with sites 27 and 67 as nearby sites, followed by taking site 27 to be the site predicted and then using site 67 as the site predicted. Using this method, we get a single large dataset whose size is equal to the sizes of datasets generated for each site in a RWIS group separately added together. By predicting the first class seen in the dataset we perform a site independent evaluation of the HMM and focuses on prediction accuracy in a group. The bigger dataset was generated for each group using the temperature data from the years 2002 and 2003. We used this dataset was used to predict the first class value (which is the predicted site) seen in the class string using HMMs and the model was evaluated using ten 10-fold cross validation runs. Results The distance between actual and predicted class value was used to evaluate the performance of the model. Figure 4.9 shows the percentage of instances (with an instance being an hourly reading) with a certain distance between the actual and predicted value, F for each of the three RWIS sets used. The percentage values were obtained after averaging the values reported for each cross-validation run. Discussion Comparing the results of Experiment 7 with this experiment, we observed (see Figures 4.8 and 4.9) that the percentage of instances with a certain distance is almost the same, that is the the results obtained for a group using a combined dataset and from averaging results from within a group where each site was trained using its dataset are about the same. From this we conclude that using a single model built from the combined dataset values for any site in a group can be can be predicted with accuracy comparable to that of the overall group. The percentage of instances predicted with a distance of 3 or less in the groups 1, 2 and 3 were 99.76%, 99.71% and 97.88% respectively, which covers almost the entire data. Thus, we can say that when the distance between the actual temperature class and the predicted class is more than three then there is a malfunction in the RWIS temperature sensor. 4.2.2 Predicting Precipitation Type This section describes the experiments performed using HMMs to predict the class value of the precipitation at an RWIS site. 4.2.2.1 Experiment 9: Predicting precipitation type using HMMs. In this experiment we predict the precipitation type reported at an RWIS site. The precipitation type information from the RWIS sites in a group are used to form the feature symbols. The results obtained from predicting temperature class were analyzed. Methodology We performed ten 10-fold cross validations using HMMs, to predict the precipitation type at an RWIS site. Years 2002 and 2003 were selected for generation of the data as these where the most recent years for which we had entire yearly data for the RWIS sensors when this experiment was conducted. The dataset used for prediction contained precipitation type information for the years 2002 and 2003 with each instances consisting of hourly class strings for a particular day. The class string in the dataset instance was obtained from appending the precipitation type of RWIS site used for prediction and from the nearby sites, which were arranged according to the distance from the site used for predictions. We cannot use the distance between actual and predicted value to be used as a measure of performance evaluation, as each precipitation type reported is unique and has a specific meaning. We observed that some RWIS sites report precipitation type as present or not present while the remaining sites indicate the type of precipitation when present. In order to compare the performance across all sites, any form of precipitation occurring was taken as precipitation present, averaged over all sites present in a set. Figure 4.10 shows the percentage of instances in the dataset that were correctly predicted as precipitation present or as not present, along with the percentage of instances where no precipitation and precipitation present were reported in the actual data. These values for each of the 13 RWIS sites on which prediction was done is shown in Appendix C. Discussion From the percentage values in Figure 4.10 and Appendix C we can say that HMM fails to predict precipitation when actually present for most of the sites with not even half of the instances that report precipitation being correctly classified. For predicting absence of precipitation most of the sites had poor values but some showed better results. It can be seen from the values in Appendix C that the percentage of instances correctly classified varies with respect to a site. We see that there are a large number of instances where the absence of precipitation was reported as precipitation and as these are seen for large number of instances, this can be seen as a factor which reduces the accuracy of the model. We find that there are also a lot of misclassifications occurring during prediction of precipitation type values. From the these results we can conclude that HMM trained using the methodology described is not a good option for predicting precipitation and thus cannot be used for detection of RWIS precipitation sensor malfunctions. This may be due to the training methodology we had used in our HMM. Overall Conclusions We compare the values predicted by the various ML methods to the actual values observed at an RWIS sensor to detect sensor malfunctions. Accuracy of the algorithm in predicting values plays a major role in determining the accuracy with which malfunctions can be identified. From the experiments performed to predict temperature at an RWIS sensor we found using current and previous three hours temperature readings and the temperature at an RWIS site in the dataset, the classification algorithms LMS and M5P gave results accurate to ±1°F and had low standard deviation across sites. LMS and M5P were identified to be able to detect sensor malfunctions accurately. The algorithms RBF Networks and CR failed to predict temperature values. The use of precipitation as an additional source of information had no significant changes on the accuracy of the algorithms. A threshold distance of 2 between actual and predicted class value was identified to be able determine sensor malfunctions when using J48 to predict temperature class values. The J48, Naive Bayes and Bayes showed varied results when predicting presence or absence of precipitation. A combination of J48 and Bayes Nets was identified to be able to detect malfunctions of a senor when predicting precipitation type. Visibility was best classified using M5P. An issue in using visibility is that RWIS sensor measures sensors to a maximum of 1.09 miles and all distances above as taken as 1.09 miles, this prevents the algorithms from capturing changes that happen beyond this point. HMMs were found to be effective in classifying temperature class but failed to predict precipitation type information. A threshold distance of 3 was identified to accurately identify sensor malfunctions. When predicting temperature class using HMM, a single model of a group obtained from using the combined dataset gave similar accuracy when compared to average of accuracies from predicting temperature at single sites over a group. From this we conclude that use a single model, built from using datasets of all sites together, can be effectively used to identity malfunctions at any site in the group. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Figure 4.10: The percentage of instances with precipitation present and with no precipitation present predicted correctly using classification algorithms.