Ferguson on Quantitative Research Methods in Planning: A Comparative Assessment of ‘Teaching’ versus ‘Practice’

Table of Contents

Abstract…………………………………………………………………………………. 

Introduction……………………………………………………………………………. 

Data and Research Methodology………………………………………………… 

Results…………………………………………………………………………………… 

Global QRM Changes……………………………………………………………….. 

Skill Group Changes………………………………………………………………… 

Regression Analysis…………………………………………………………………. 

Discussion……………………………………………………………………………… 

Teaching versus Practice…………………………………………………………… 

Teaching and Practice………………………………………………………………. 

Conclusions……………………………………………………………………………. 

Acknowledgements…………………………………………………………………. 

References…………………………………………………………………………….. 

Abstract

Quantitative research methods (QRM) are important in planning practice and education. Data from three landmark QRM studies are pooled and analyzed using appropriate statistical procedures. Applying a simple difference of proportions test to the combined data naively suggests that QRM use increased significantly in both planning practice and education between 1974 and 1986. Both decreased significantly between 1986 and 1992. A slightly more sophisticated technique reveals that QRM use in planning was much more stable between 1974 and 1992 than the naïve test might otherwise suggest. The only major increase in QRM use between 1974 and 1992 seems to involve the introduction of computing and related information technologies into the planning workplace. Research, statistics, data collection, and forecasting are the most important QRM teaching tools. Evaluation, data collection, implementation, and forecasting are the most important applications of QRM in planning practice. Relevant conclusions derive as appropriate from these inferential findings.

Introduction

Quantitative research methods (QRM) arguably are important in planning, but perhaps not strictly speaking necessary. QRM are after all merely a small subset of the professional skills required in planning practice, and not necessarily the most important subset either. Schon et al (1976) showed that a variety of writing, communication, and management skills were much more important than most analysis skills in planning practice. Krueckeberg (1978) demonstrated that the decision rules employed by Schon et al (1976) to prove design skills superior to analysis skills as well were fundamentally flawed in their construction.

Krueckeberg (1978) reviewed a quarter century of planning literature to cull 20 surveys on the use of quantitative methods and models in planning practice and education. Dandekar (1986) argued that qualitative methods are equally important in planning practice and education, without providing any direct evidence (at least not of a quantitative nature) in support of this bold assertion. Nevertheless, it is a commonly held belief among planners that qualitative methods are just as important as quantitative ones in professional planning practice, if indeed not more so.

Pearlman (1983) indexed a quarter century of JAPA articles by subject, title, and author. Of 50 articles listed under “methods and models”, over half (29) are devoted to the subject of modeling. These highly variable and sophisticated modeling exercises are followed at a very large (and some might even say growing) semantic distance by forecasting (7), evaluation (6), data collection (4), statistics (2), implementation (1), and computers (1). Many similar (as well as some different) methods are located under many other headings in Pearlman (1983), but the force and intent of what most people mean when they talk about “planning methods” is made abundantly clear by this breakdown.

QRM may in fact include some or all of the following general types of more or less numerically inclined planning skills:

• Computers (spreadsheet and database applications; planning and geographic information systems; computer languages and programming)
• Data collection (focus groups, surveys and sampling; Census and other secondary data; literature reviews and other sources of tertiary data)
• Forecasting (population, employment, and housing; urban growth and economic development)
• Modeling (land use and activity systems; transportation systems and urban travel behavior; simulations, systems analysis, and operations research)
• Evaluation (fiscal, economic, community, development, neighborhood, social, environmental, ecological, and political impacts)
• Implementation (programming, budgeting, and scheduling; decision making; performance monitoring and ex-post evaluation)
• Mathematics (algebra, trigonometry, geometry, calculus)
• Statistics (descriptive, inferential, and multivariate statistics; regression, factor, and cluster analysis)
• Research methods (experimental design, hypothesis testing)
• Qualitative methods (scenario construction; content and issues analysis)

The objective of this paper is to determine the relative priority of each of these general types of QRM skill groups in planning practice and education through the development of inferential statistics to test relevant data in the planning literature.

Data and Research Methodology

The data used in this analysis come from three landmark studies of QRM in planning, namely:

• Isserman (1975) and Isserman (1977): two independent but related studies, one of planning schools conducted in 1973-74, the other of planning practitioners conducted the following year. Isserman evaluated 22 different methods representing six separate and distinct types of analysis skills: computers, data collection, evaluation, forecasting, modeling, and statistics.

• Contant and Forkenbrock (1986): this follow-on study to Krueckeberg evaluated an expanded list of 28 different methods representing eight analysis skill groups. Nineteen of Isserman’s methods were retained, three were dropped, and nine more were added. To Isserman’s six analysis skill groups two more were added: implementation and qualitative methods.

• Kaufman and Simons (1995): this follow-on study to Contant and Forkenbrock evaluated an expanded list of 53 different methods representing 10 analysis skill groups. Ten of Isserman’s and 17 of Contant and Forkenbrock’s methods were retained basically unchanged, while many others were renamed, modified, split, or simply added to the list. To Contant and Forkenbrock’s eight analysis skill groups two more were added: mathematics and research methods

Sample sizes in all three studies were approximately equivalent, with 41-62 schools and 69-106 practitioners represented. The populations studied were quite similar as well. The sampling procedures used and methods tested varied from one study to the next, but are roughly comparable when suitable caveats are employed (Table 1).

Results

Global QRM Changes

Of the 22 methods tested by Isserman in 1974, five are roughly comparable and 14 others are exactly comparable across all three studies. Roughly comparable methods include variations on a single theme such as “cost-revenue analysis” and “financial analysis.” Exactly comparable methods include minor labeling changes such as “non-linear programming” and “nonlinear programming.”

A small but significant increase in the practical application of QRM occurred between 1974 and 1986, followed by a somewhat larger and more significant decrease between 1986 and 1992. A large and highly significant increase in QRM teaching occurred between 1974 and 1986. There was no change in QRM teaching between 1986 and 1992, at least when controlling for roughly or exactly equivalent methods (Table 2).

The use of QRM increased in both practice and education between 1974 and 1986, and both of these increases are real. The observed decrease in the practical application of QRM between 1986 and 1992 is probably a statistical artifact associated with less restrictive sampling and a corresponding diminution of methodological awareness as opposed to actual usage.¹

The observed increase in QRM teaching between 1974 and 1986 is large and statistically significant. Observed changes in the practical application of QRM are statistically significant, but not particularly large. It appears in fact that the practical use of QRM was remarkably stable between 1974 and 1992, at least at the global scale of analysis.

Skill Group Changes

Between 1974 and 1986, major increases in teaching QRM occurred in four of Isserman’s six analytical skill groups: computers, data collection, evaluation, and statistics. The percentage of planning schools with computers and evaluation as required subjects both doubled during this time. Two analytical skill groups showed no significant increase in QRM teaching, forecasting and modeling. Only computers experienced a truly large increase in the practical application of QRM between 1974 and 1986 (Table 3).

QRM teaching was remarkably stable between 1986 and 1992. The sole significant change (implementation) is probably a statistical artifact due to survey construction, rather than a real change.²  The decrease in QRM practice observed between 1986 and 1992 is a general decline across the board, even to include computers. This pattern suggests another statistical artifact associated with sample design, rather than a real decrease in practical QRM use. The basic pattern across all three studies reveals much more stability than change. The only truly major increases occurring in the teaching of computers and several other closely related skills between 1974 and 1986, with no major changes observed between 1986 and 1992, other than those associated with sample design.³ Kaufman and Simons probably should have waited a few more years to perform their study. Alternatively, they might have settled for less major changes in sampling procedures and QRM definitions in order to produce more reliable statistical results.

Regression Analysis

Pooled data from all three studies were analyzed using regression (Table 4). The results reconfirm and help clarify the previously discussed global and analysis skill group changes. According to the model, there was no significant change in QRM teaching or practice between 1986 and 1992. The observed increase in teaching and practice between 1974 and 1986 is less significant in the model than in the previously reported direct comparisons. This further reinforces the idea that QRM practice and education are both largely characterized by stability rather than change, especially once methodological differences among the three studies are properly controlled.

The number of categories used to describe particular skill groups was significant in the practice equation, but not in the teaching equation. This may reflect a difference in attitudes toward surveys and semantic scales among academics and practitioners. The longer the list of methods tested, the easier it may be to exhaust the patience of practitioners, who become more selective in answering questions sooner than do academics. It may also reflect variable conditions of methodological awareness. Academics presumably are more familiar with a broader range of more specialized skills and techniques, especially when the academics in question are those in charge of QRM instruction, and therefore specialists in that particular aspect of planning education and research.

Discussion

Teaching versus Practice

QRM skill groups are ranked by regression model coefficients in Table 5. Data collection appears high atop both lists. Research and statistics are high on the academic list. This is no surprise, although it should be pointed out for clarity that the situation today is much different than it was thirty years ago. Back in 1974, most academic units in planning were dominated by former planning practitioners, rather than the more research-oriented, publication-producing, doctorate-holding planning academics of today. Evaluation and implementation are high on the practitioner list. This too is not a major surprise, as planning practice has moved from pure visioning processes towards more results-oriented approaches in the last few decades.

Mathematics and qualitative methods are low on both lists. Basic mathematical skills are usually a prerequisite for statistics, forecasting, and other more advanced quantitative methods. Qualitative methods may be undervalued (at least on the academic side), particularly since the person responsible for quantitative methods instruction was the one asked to complete the survey. Qualitative methods often are the responsibility of a completely different person, and are most probably underrepresented here.

Teaching and Practice

QRM model coefficients are added, subtracted, and normalized for comparison purposes in Table 6. When added together, these represent a proxy for relevance to the profession, balanced evenly between present practical and educational requirements. When subtracted, these represent unmet teaching needs (a look towards the future). Data collection, forecasting, statistics, and evaluation rank highest overall in terms of their current relevance to the profession. These are mainly positive skills. Implementation, evaluation, qualitative methods, and modeling rank highest in terms of unmet future teaching needs. These are mainly normative skills. The likely trend of the past through tomorrow seems reasonably clear based on these results.

Skills in which planning academics currently leads the way include the following:

• Forecasting – population projections, employment forecasting, and time series analysis form the fundamental core of planning sciences, but have declined slightly in importance over the last several decades

• Computers – computer applications have proliferated in recent years, while computer programming is an increasingly specialized sub-field, of major importance only to a small cadre of technical experts who are interested in that sort of thing; most other methods, including both quantitative and qualitative methods, increasingly rely on computers to provide the data and analysis tools needed for success in planning practice and education; even more than population and employment forecasting, the ability to interact with a computer has become, if not the pinnacle of a planning education, at least a firm foundation for its future

• Statistics – a relatively rapid increase in the early years of this study period, followed by a recent platform of stability; regression analysis may be overemphasized to some degree, while other important statistical techniques, such as factor and cluster analysis, are often benignly neglected; statistics are closely connected to planning research and publication; statistics may be less important to planning practice than education, but remain critical to any scientific progress in the field

• Research – a new addition to the 1992 study, research methods may be nothing more than a synonym for more qualitative approaches; the need for research in planning academia is obvious; research may be less valued in planning practice, but this is not necessarily a good thing; without practical applications of planning research in planning practice, it is not clear how further progress in the field can be made

Skills in which planning practitioners have the lead include the following:

• Implementation – there is a continuing problem of definition in terms of what is meant by “implementation”; scheduling and budgeting are management tools, while decision making, communication, and performance monitoring are related aspects of plan implementation not included in any of the three landmark studies considered here; more work needs to be done in defining implementation in such a way that it can be understood and appreciated, especially from a planning research perspective

• Evaluation – economics seems to be overemphasized, while politics is clearly underrepresented in the evaluation methods included in all three studies; cost-benefit analysis is included, while cost-effectiveness analysis is excluded from all three; economic impact and cost-revenue (or financial) analysis are included, while social and environmental impact analysis are excluded from all three; multi-criteria analysis is excluded from Contant and Forkenbrock, and mislabeled (or over-theorized) and therefore misunderstood (and thereby underrepresented) in Krueckeberg (game theory)⁴ and Kaufman and Simons (multiattribute utility models, decision analysis); economics is an important aspect of evaluation, but it is not the only way of looking at the world in terms of assigning values to alternative outcomes, including most particularly the choice of method employed in assigning such values in the first place

• Qualitative – qualitative methods may inform half or more of planning practice, not to forget planning research, publications and scholarship; the token representation of qualitative methods in two of three QRM studies cannot be taken seriously as adequately representative of its importance to the field, even ignoring the obvious problem with the sampling procedures used in all three studies

• Modeling – unlike the three previous skill groups, modeling is a much more concrete subject and positive planning skill, and is more than adequately represented in the QRM data; the poor showing of modeling skills in the survey results from all three studies cannot be a coincidence, but this should not be exaggerated in terms of its overall importance either; modeling is the single most advanced and perhaps the most specialized of all analysis skills in planning; it is therefore understandable that most modeling skills are not taught as required subjects to all planning students; modeling is more likely these days to be taught as an elective subject, or included in specialization courses associated with particular subcategories of planning such as land use or transportation; the sole exception to this rule is gravity modeling, which is much more frequently taught as a required subject than any other single modeling technique, even though it is far from being the type of model most frequently encountered in planning practice; all of the analysis skills which appear high on the list of future unmet teaching needs may simply be covered in other ways than a required QRM course, in which case it could be argued that all methods are adequately covered in most planning curricula today, a most sanguine consideration

Skills that balance evenly between planning practice and education at present include the following:

• Mathematics – algebra and calculus are prerequisites for entry into many graduate planning programs; a review of basic mathematical concepts may be helpful to students who are mathematically challenged or otherwise ill-prepared by their previous training and education to tackle statistics and related “hardcore” quantitative methods in a graduate degree program

• Data collection –this skill is very important to both teaching and practice, and is therefore adequately balanced as well; the most important QRM skill to the planning profession as a whole is in fact data collection; with the advent of computers, spreadsheets, and other information technologies, database management is an increasingly important concomitant skill; regardless of what it may be used for, planners increasingly have access to and are expected to use computerized data in the daily performance of their jobs; given this increasing expectation, it is important for planners to understand the needs and requirements of data collection, if only so that they will know how and when to collect original data in support of their various personal and professional causes; this as opposed to relying on less expensive and usually more timely secondary or tertiary data as the planning instruments of first choice, if and as this may be deemed appropriate to the situation at hand

Conclusions

Is reliance on QRM in planning practice and education increasing or decreasing? It appears that in fact QRM supply and demand are both remarkably stable, especially since 1986. Other than a marked increase in the use of computers, indications of truly fundamental changes in QRM use in planning practice and education over the last quarter century simply are not present anywhere in the data. Evaluation, data collection, and statistics all increased their relative presence in the required core of planning education between 1974 and 1986. This may have been an indirect response to new opportunities created by improved information technologies, rather than the direct effect of increased demand for QRM in the marketplace for professional planning skills.

Is the supply of and demand for QRM in planning practice and education balanced or unbalanced? Krueckeberg showed that supply and demand were unbalanced if only required methods were included, but properly balanced when electives entered the equation. QRM educational requirements increased significantly between 1974 and 1986, but did not change between 1986 and 1992. No recent studies have appeared, but a review of the planning literature over the last 10 years shows no particular increase in the news value of QRM. If anything, it is rather more likely that the reverse is true.

Does planning education lead or lag planning practice in determining future QRM needs and requirements? Surprisingly, despite frequent laments in the planning literature to the contrary concerning too much emphasis on QRM and not enough on communicative practice, it appears that education definitely lags practice in this respect. It may be remarked in passing that well over half of all planning practitioners surveyed in 1974 identified knowledge of computers as integral to an adequate educational preparation for a career in planning. Fewer than half of all planning programs required computing, however, as part of their graduate planning degree programs in that year.

Should QRM be required of all planning students, offered as an alternative among planning electives, or provided as a service course by other departments? The answer, of course, is all of the above, as Krueckeberg pointed out in his 1978 survey of the field. The QRM “core” in planning education currently includes research methods, statistics, data collection, and forecasting techniques. Computing is universally required, but no longer operates as an independent subject.  Rather, it provides the sturdy platform on which most other methods can and do rest. Evaluation (especially cost-benefit analysis), modeling (particularly the gravity model), and implementation (whose role in QRM remains problematical, i.e., poorly defined) are somewhat less critical, or at least less commonly placed in the required core of planning.

How important are specific types of QRM in planning practice? The available data do not allow any substantive discussion of this issue at this time. Previous studies were more interested in the academic implications of planning practice than the practical implications of a planning education. Awareness of methods that may occasionally crop up in planning practice is not nearly the same as assigning concrete values to those methods, or even the frequency with which such methods might actually come into play. The data discussed in this paper provide a more than adequate reflection of the role of QRM in planning education over the last quarter century, but the same cannot realistically be said about QRM in planning practice. This is an area for possible future research.

Are any particularly important quantitative methods systemically excluded from the data? The only major topics that seem overly neglected are non-economic evaluation methods and non-regression statistical analysis. This is a reflection of current built-in biases in planning academia, which reflect its origins in terms of the disciplines called upon most frequently in earlier years to assist in the professionalization of the planning academy. This “academization” process clearly reflects major contributions from both urban economics and regional science. Political science, psychology, and sociology played much smaller roles in that process, and make less significant contributions to the specific types of QRM considered relevant to planning practice and education today. Whether this is entirely appropriate remains an open question at this time.

Acknowledgements

The author would like to thank Andrew Isserman of the University of Illinois, Urbana-Champaign, Sanda Kaufman of Cleveland State University, and Tim Chapin of Florida State University for assistance with the data, survey methods used, and the development of the conceptual framework on which this analysis is based.

References

Contant, C.K. and Forkenbrock, D. 1986. Planning methods: an analysis of supply and demand. Journal of Planning Education and Research 6: 10-21.

Dandekar, H. 1986. Some uses and potentials of qualitative methods in planning. Journal of Planning Education and Research 6: 42-9.

Isserman, A.M. 1975. Quantitative methods courses in graduate programs of urban and regional planning. Bulletin of the Association of Collegiate Schools of Planning 13(3): 8-11.

Isserman, A.M. 1977. Planning practice and planning education: the case of quantitative methods. Bulletin of the Association of Collegiate Schools of Planning 15(1): 1-7.

Jefferson, R. 1973. Planning and the Innovation Process. Oxford: Pergamon.

Kaufman, S. and Simons, R. 1995. Quantitative and research methods in planning: are schools teaching what practitioners practice? Journal of Planning Education and Research 15: 17-33.

Krueckeberg, D.A. 1978. Practical demands for analytical methods. In Burchett, R.W. and Sternlieb, G., eds., Planning Theory in the 1980s, pp. 309-40, Brunswick, NJ: Centre for Urban Policy Research.

Pearlman, K., ed. 1983. Cumulative index of volumes 24 through 49. Journal of the American Planning Association 49(4): 385-512.

Schon, D., N.S. Cremer, P. Osterman and C. Perry. 1976. Planners in transition: report on a survey of M.I.T.’s Department of Urban Studies, 1960-71. Journal of the American Institute of Planners 42(2): 193-202.

  

1. Contant and Forkenbrock queried planning directors on the use of methods among all of the planners within their department or division. Kaufman and Simons queried planning practitioners on the use of methods within their localized work unit, arguably a much smaller unit of measurement than an entire department or division. Both studies surveyed AICP members, but Contant and Forkenbrock’s restricted sample of planning directors presumably covered a larger number of individual planners with more varied work responsibilities and broader methodological needs and requirements. This then becomes an issue of variable awareness rather than substantive difference. [↩]
2. “Scheduling” was included as a QRM in both 1986 and 1992. Two “new” implementation methods appear in the 1992 study, “budget preparation” and “capital improvement plans.” These overlap with “scheduling” to a considerable extent, and may have sown some minor semantic confusion among the respondents to the 1992 survey. [↩]
3. It seems obvious that computers have facilitated QRM teaching in many different and sometimes quite profound ways: statistical software, spreadsheet applications, and database management programs are just a few of the more important examples of this earthshaking phenomenon. [↩]
4. Technically, “game theory” is Isserman’s term for something similar to multi-criteria evaluation. Krueckeberg himself identifies a British study (Jefferson 1973) in which cost-effectiveness, planning balance sheets (a variation on cost-benefit analysis) and goals achievement matrices (an earlier name for multi-criteria analysis) are specifically included as alternative types of evaluation methods in planning. [↩]