I represent Industrial Wind Action Group, a national organization focused on raising awareness of the negative impacts of utility-scale wind if sited improperly. In this capacity, our organization closely monitors wind energy proposals, development, and post-construction performance and attendant impacts. We are very pleased to have this opportunity to submit our comments on your research.
We read your report carefully paying particular attention to the methodology employed to arrive at your conclusions. Our comments are outlined below.
1. Regression analysis is not in accordance with the International Association of Assessing Officers' (IAAO) established methods.
We are concerned at the outset that the analysis employed is not in accordance with the International Association of Assessing Officers' (IAAO) established methods for measuring the accuracy and reliability of regression models [1] for the purposes of estimating real estate values and certifying those models. We are unaware of any other recognized certification schemes.
The study appears to make no distinction between regression as a recognized tool used by statisticians to estimate a relationship between dependent and independent variables and that of hedonic analysis which is an interpretive technique used to evaluate the results of a regression.
2. No clear evidence the data used was checked for accuracy.
The report states it included "valid sales" in the data set, however we can find no reference to what the authors consider a valid sale. While we expect the data set to include only sales which are arms-length, we cannot determine from the report whether the local officials who assisted in gathering the data included or excluded the below categories of properties. [2] In many cases, sales that would fall into the below categories are difficult to ferret out and may require investigation beyond looking at available real estate transfer records.
a) Short sales where buyers cooperated with lenders to sell a property;
b) Foreclosure sales or Sheriff sales where bidders may be denied access to the inside of a property to evaluate possible deficiencies. Bidders may also risk buying second and third mortgages;
c) Real estate owned or lender sales where the lender has already taken possession of the property and is now the owner. These sales may also include 'deed in lieu of foreclosure' transactions;
d) Fannie Mae, VA, and Federal Housing Administration sales which are marketed in the MLS along with normal sales;
e) Flipping of properties involving purchase and quick sale which can artificially drive up the values of homes in an area;
f) Other auction sales.
The impact of just a handful of invalid sales can skew large data sets as was found in the case study in the Longmont area of Colorado. [3] In that situation, just fifteen "dirty sales" adversely and significantly affected the mass appraisal results for approximately 26,000 residential properties.
3. No information in the study confirms whether the model was tested or calibrated using actual sales data.
According to IAAO, when a model is specified an iterative process of calibrating the model using data sets is necessary to test and fine tune the model's coefficients. In regression, there is no means otherwise of knowing whether one regression is better than another, except by measuring how well the model estimates sales prices. Thousands of possible real estate regression models can apply to any given situation. If the model had been calibrated, we recommend the authors explain in the report the process followed.
4. Model is not peer-reviewed. Data withheld from independent reviewers.
While the authors have suggested this study will be peer-reviewed they have refused to release any of the data set to reviewers. [4] Any market analysis must be prepared with an expectation that the results can be defended and/or the process duplicated by independent reviewers. Absent access to the data, third party reviewers cannot test the methodology. As long as the data set is withheld from the public we believe it would be inappropriate to characterize this study as being 'peer-reviewed'.
5. The data set is not homogeneous; data is drawn from across the country.
A basic assumption of a regression is that the database is reasonably homogeneous. Homogeneous means that the housing is similar in market characteristics such as approximate size, age, quality, available amenities (schools, shopping, security, access to work and recreation, etc) and are examined in a similar economic setting (employment, availability and cost of financing, market growth or decline and the like), among other factors. Homogeneity of the marketplace is fundamental to regression analysis and it is well documented regression techniques are difficult to utilize on data sets that vary substantially due to differing characteristics. In this study, data was included from nine different states across the United States making similarities and differences impossible to properly assess.
The data set selected by the authors includes 4,895 "valid sales" of which a subset of property characteristics is identified and then averaged to produce a composite home of 47 years in age with 1,628 square feet of finished living area above ground, 1.75 bathrooms situated on 1.09 acres and having an average condition. The data shows home sale prices ranging from as low as $10,492 to as high as $647,500.
The variation in house price alone indicates a failure to meet the requirement of homogeniality. But the problem is more pervasive. If we look at the 'age' characteristic in Table 4 of the report, average age of the home at the time of the sale was 47 years with a standard deviation of 36. In other words, within one standard deviation, 68.2% of the homes in the data set range in age from 11 years to 83 years. We have no way of knowing how age is influencing sale prices within the study's data set. A similar concern is raised with regard to the square footage of the home, number of baths, etc.
In addition, since the data set is drawn from diverse locations across the country, we cannot be certain that the coefficients used in the hedonic analysis are appropriate for each location. For example, fireplaces or finished basements in Texas may be perceived as less valuable than central air conditioning and the reverse may be true for the same characteristics found in homes in upstate New York.
6. The data set omits property characteristics.
A host of property characteristics are omitted making it impossible to segment out the influence of these characteristics despite their omission. We cannot understand why the study looked at exterior finish but omitted the number of bedrooms or the availability of a garage.
The Hedonic analysis method argues that the coefficients of the regression may be quantitatively interpreted as the marginal contribution of specific independent variables to the sale price. If a coefficient is accurate it should reflect only the contribution of the specific variable to the sale price. However, when variables are omitted from the model, such as number of bedrooms, the effect may be to inflate the size of the other coefficients by the omitted variable's contribution. [5] There is no way to know the effect of not including number of bedrooms in the model unless the authors rerun the analysis with that variable included.
The authors argue that despite some omitted variables, the restricted model performs well producing an adjusted R2 of 0.78 and that adding additional variables did not significantly improve the model's performance. The IAAO would disagree with the authors' conclusion. The IAAO standard for regression models states the estimate-to-sale price "should be within 5 percent of the overall estimate-to-sale ratio for all strata; and the overall estimate-to-sale level should be within 10 percent of the desired level of 100 percent. An R2 of 0.78 is not good enough according to the IAAO.
7. Study neglects to explain the risks of employing Hedonic analysis.
Causal conclusions drawn about a data set when utilizing hedonic analysis are often times unsupportable. Neter, Kutner et.al. [6] state:
"The existence of a statistical relation between the response variable Y and the explanatory or predictor variable X does not imply in any way that Y depends causally on X. No matter how strong is the statistical relation between X and Y, no cause-and effect pattern is necessarily implied by the regression model. ... Regression analysis by itself provides no information about causal patterns and must be supplemented by additional analyses to obtain insight about causal relations. ... A major limitation of observational data is that they often do not provide adequate information about cause-and effect relationships."
This is just one of several flaws identified in the literature, yet the authors of this study neglect to describe any of the pitfalls of hedonic analysis. While the study appears to advocate for hedonic analysis we would expect the authors to make at least some attempt to defend their reliance on this method in light of literature which is highly critical, in fact dismissive, of the methodology used.
8. Background review of other studies.
All of the studies mentioned in Section 2 under 'Previous Research' have limitations, and in some cases fatal flaws which render their results misleading and perhaps invalid. Study author Ben Hoen captured some of these issues in his 2006 report on the Fenner wind project. We recommend that this study, at a minimum, detail what is already known about the existing studies to assist those reviewing the literature in assessing prior works. We see no point in perpetuating the results of studies that are known to be flawed.
9. Closing Comments
We have other areas of concern with the current study but the main issues relate to the basic analysis employed.
Since the study attempts to use a recognized method of appraising properties (regression), we believe the authors would have benefited by having a certified appraiser and member of IAAO on their team at the outset. While appraisers may be reviewing this draft report, the study is largely complete and it's likely too late to make the corrections necessary to correct for the problems.
We believe the flaws we found in the methodology render the results of this study meaningless. More to the point, should an expert witness rely on this study to argue property values are not diminished by proximity to industrial scale turbines, it is likely a qualified appraiser with experience in regression techniques and the problems of hedonic analysis will counter such assertions.
Thank you very much for permitting us to review and submit comments on this important study. I hope you find our comments useful and look forward to being kept updated on your continued progress.
Lisa Linowes
Industrial Wind Action www.windaction.org
__________________________________
[1] International Association of Assessing Officers. Sep 2003. Standard on Automated Valuation Models (AVMs) http://www.iaao.org/uploads/AVM_STANDARD.pdf
[2] Liflander, John. Jun 2000. Foreclosure - The Looming Threat to Property Values. Fair & Equitable.
[3] Cholvin, Brooke and Simpson, Danielle. Aug. 2009. Assessing Mortgage Fraud. Fair & Equitable.
[4] E-mail correspondence between Ben Hoen and Lisa Linowes, Sep 8, 2009.
[5] Wilson, Albert R. Summer 2006. Real Property Damages and Rubber Rulers. Real Estate Issues.
[6] John Neter, Michael Kutner, Christopher J. Nachtsheim, William Wasserman, Applied Linear Regression Models, Third Edition (Chicago: Irwin, 1996).
No comments:
Post a Comment