Scenario-based projections of social processes

As we have noted in previous posts, social outcomes are highly path-dependent and contingent (link, link, link, link). This implies that it is difficult to predict the consequences of even a single causal intervention within a complex social environment including numerous actors -- say, a new land use policy, a new state tax on services, or a sweeping cap-and-trade policy on CO2 emissions. And yet policy changes are specifically designed and chosen in order to bring about certain kinds of outcomes. We care about the future; we adopt policies to improve this or that feature of the future; and yet we have a hard time providing a justified forecast of the consequences of the policy.

This difficulty doesn't only affect policy choices; it also pertains to large interventions like the democracy uprisings in the Middle East and North Africa. There are too many imponderable factors -- the behavior of the military, the reactions of other governments, the consequent strategies of internal political actors and parties (the Muslim Brotherhood in Egypt) -- so activists and academic experts alike are forced to concede that they don't really know what the consequences will be.

One part of this imponderability derives from the fact that social changes are conveyed through sets of individual and collective actors. The actors have a variety of motives and modes of reasoning, and the collective actors are forced to somehow aggregate the actions and wants of subordinate actors. And it isn't possible to anticipate with confidence the choices that the actors will make in response to changing circumstances. At a very high level of abstraction, it is the task of game theory to model strategic decision-making over a sequence of choices (problems of strategic rationality); but the tools of game theory are too abstract to allow modeling of specific complex social interactions.

A second feature of unpredictability in extended social processes derives from the fact that the agents themselves are not fixed and constant throughout the process. The experience of democracy activism potentially changes the agent profoundly -- so the expectations we would have had of his/her choices at the beginning may be very poorly grounded by the middle and end. Some possible changes may make a very large difference in outcomes -- actors may become more committed, more open to violence, more ready to compromise, more understanding of the grievances of other groups, ... This is sometimes described as endogeneity -- the causal components themselves change their characteristics as a consequence of the process.

So the actors change through the social process; but the same is often true of the social organizations and institutions that are involved in the process. Take contentious politics -- it may be that a round of protests begins around a couple of loose pre-existing organizations. As actors seek to achieve their political goals through collective action, they make use of the organizations for their communications and mobilization resources. But some actors may then also attempt to transform the organization itself -- to make it more effective or to make it more accommodating to the political objectives of this particular group of activists. (Think of Lenin as a revolutionary organization innovator.) And through their struggles, they may elicit changes in the organizations of the "forces of order" -- the police may create new tactics (kettling) and new sub-organizations (specialized intelligence units). So the process of change is likely enough to transform all the causal components as well -- the agents and their motivations as well as the surrounding institutions of mobilization and control. Rather than a set of billiard balls and iron rods with fixed properties and predictable aggregate consequences, we find a fluid situation in which the causal properties of each of the components of the process are themselves changing.

One way of trying to handle the indeterminacy and causal complexity of these sorts of causal processes is to give up on the goal of arriving at specific "point" predictions about outcomes and instead concentrate on tracing out a large number of possible scenarios, beginning with the circumstances, actors, and structures on the ground. In some circumstances we may find that there is a very wide range of possible outcomes; but we may find that a large percentage of the feasible scenarios or pathways fall within a much narrower range. This kind of reasoning is familiar to economists and financial analysts in the form of Monte Carlo simulations. And it is possible that the approach can be used for modeling likely outcomes in more complex social processes as well -- war and peace, ethnic conflict, climate change, or democracy movements.

Agent-based modeling is one component of approaches like these (link).  This means taking into account a wide range of social factors -- agents, groups, organizations, institutions, states, popular movements, and then modeling the consequences of these initial assumptions. Robert Axelrod and colleagues have applied a variety of modeling techniques to these efforts (link).

Another interesting effort to carry out such an effort is underway at the RAND Pardee Center, summarized in a white paper called Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis. Here is how the lead investigators describe the overall strategy of the effort:

This report describes and demonstrates a new, quantitative approach to long-term policy analysis (LTPA).  These robust decisionmaking methods aim to greatly enhance and support humans’ innate decisionmaking capabilities with powerful quantitative analytic tools similar to those that have demonstrated unparalleled effectiveness when applied to more circumscribed decision problems.  By reframing the question “What will the long-term future bring?” as “How can we choose actions today that will be consistent with our long-term interests?” robust decisionmaking can harness the heretofore unavailable capabilities of modern computers to grapple directly with the inherent difficulty of accurate long-term prediction that has bedeviled previous approaches to LTPA. (iii)

LTPA is an important example of a class of problems requiring decisionmaking under conditions of  deep uncertainty—that is, where analysts do not know, or the parties to a decision cannot agree on, (1) the appropriate conceptual models that describe the relationships among the key driving forces that will shape the long-term future, (2) the probability distributions used to represent uncertainty about key variables and parameters in the mathematical representations of these conceptual models, and/or (3) how to value the desirability of alternative outcomes. (iii)

And here, in a nutshell, is how the approach is supposed to work:

This study proposes four key elements of successful LTPA: 

• Consider large ensembles (hundreds to millions) of scenarios.
• Seek robust, not optimal, strategies.
• Achieve robustness with adaptivity.
• Design analysis for interactive exploration of the multiplicity of plausible futures. 

These elements are implemented through an iterative process in which the computer helps humans create a large ensemble of plausible scenarios, where each scenario represents one guess about how the world works (a future state of the world) and one choice of many alternative strategies that might be adopted to influence outcomes. Ideally, such ensembles will contain a sufficiently wide range of plausible futures that one will match whatever future, surprising or not, does occur—at least close enough for the purposes of crafting policies robust against it.  (xiii)

Thus, computer-guided exploration of scenario and decision spaces can provide a prosthesis for the imagination, helping humans, working individually or in groups, to discover adaptive near-term strategies that are robust over large ensembles of plausible futures. (xiv)

The hard work of this approach is to identify the characteristics of policy levers, exogenous uncertainties, measures, and relationship (XLRM).  Then the analysis turns to identifying a very large number of possible scenarios, depending on the initial conditions and the properties of the actors and organizations. (This aspect of the analysis is analogous to multiple plays of a simulation game like SimCity.) Finally, the approach requires aggregating the large number of scenarios to allow the analysis to reach some conclusions about the distribution of futures entailed by the starting position and the characteristics of the actors and institutions.  And the method attempts to assign a measure of "regret" to outcomes, in order to assess the policy steps that might be taken today that lead to the least regrettable outcomes in the distant future.

It appears, then, that there are computational tools and methods that may prove useful for social explanation and social prediction -- not of single outcomes, but of the range of outcomes that may be associated with a set of interventions, actors, and institutions.

Read more »

Posted in: agency,analytical sociology,CAT_methodology,localism,prediction

The inexact science of economics

Image: social accounting matrix, Bolivia, 1997

Economics is an "inexact" science; or so Daniel Hausman argues in The Inexact and Separate Science of Economics (Google Books link).  As it implies, this description conveys that economic laws have only a loose fit with observed economic behavior.  Here are the loosely related interpretations that Hausman offers for this idea, drawing on the thinking of John Stuart Mill:

1. Inexact laws are approximate.  They are true within some margin of error.
2. Inexact laws are probabilistic or statistical.  Instead of stating how human beings always behave, economic laws state how they usually behave.
3. Inexact laws make counterfactual assertions about how things would be in the absence of interferences.
4. Inexact laws are qualified with vague ceteris paribus clauses. (128)

Economics has also been treated by economists as a separate science: a science capable of explaining virtually all the phenomena in a reasonably well-defined domain of social phenomena.  Here is Hausman's interpretation of a separate science:

1. Economics is defined in terms of the causal factors with which it is concerned, not in terms of a domain.
2. Economics has a distinct domain, in which its causal factors predominate.
3. The "laws" of the predominating causal factors are already reasonably well-known.
4. Thus, economic theory, which employs these laws, provides a unified, complete, but inexact account of its domain. (90-91)

These characteristics of economic theories and models have implications for several important areas: truth, prediction, explanation, and confirmation.  Is economics a scientific theory of existing observable economic phenomena?  Or is it an abstract, hypothetical model with only tangential implications for the observable social world?  Is economics an empirical science or a mathematical system?

Let's look at these questions in turn.  First, can we give a good interpretation of what it would mean to believe that an inexact theory or law is "true"?  Here is a possible answer: we may believe that there are real but unobservable causal processes that "drive" social phenomena.  To say that a social or economic theory is true is to say that it correctly identifies a real causal process -- whether or not that process operates with sufficient separation to give rise to strict empirical consequences.  Galilean laws of mechanics are true for falling objects, even if feathers follow unpredictable trajectories through turbulent gases.

Second, how can we reconcile the desire to use economic theories to make predictions about future states with the acknowledged inexactness of those theories and laws? If a theory includes hypotheses about underlying causal mechanisms that are true in the sense just mentioned, then a certain kind of prediction is justified as well: "in the absence of confounding causal factors, the presence of X will give rise to Y." But of course this is a useless predictive statement in the current situation, since the whole point is that economic processes rarely or never operate in isolation. So we are more or less compelled to conclude that theories based on inexact laws are not a useable ground for empirical prediction.

Third, in what sense do the deductive consequences of an inexact theory "explain" a given outcome -- either one that is consistent with those consequences or one that is inconsistent with the consequences? Here inexact laws are on stronger ground: after the fact, it is often possible to demonstrate that the mechanisms that led to an outcome are those specified by the theory. Explanation and prediction are not equIvalent. Natural selection explains the features of Darwin's finches -- but it doesn't permit prediction of future evolutionary change.

And finally, what is involved in trying to use empirical data to confirm or disconfirm an inexact theory?  Given that we have stipulated that the theory has false consequences, we can't use standard confirmation theory.  So what kind of empirical argument would help provide empirical evaluation of an inexact theory?  One possibility is that we might require that the predictions of the theory should fall within a certain range of the observable measurements -- which is implied by the idea of "approximately true" consequences.  But actually, it is possible that we might hold that a given theory is inexact, true, and wildly divergent from observed experience.  (This would be true of the application of classical mechanics to the problem of describing the behavior of light, irregular objects shot out of guns under water.)  Hausman confronts this type of issue when he asks why we should believe that the premises of general equilibrium theory are true. But here too there are alternatives, including piecemeal confirmation of individual causal hypotheses. Hausman refers to this possibility as a version of Mill's deductive method.

I take up some of these questions in my article, "Economic Models in Development Economics" link, included in On the Reliability of Economic Models: Essays in the Philosophy of Economics.  This article discusses some related questions about the reliability and applicability of computable general equilibrium models in application to the observed behavior of real economies.  Here are some concluding thoughts from that article concerning the empirical and logical features that are relevant to the assessment of CGE models:

"The general problem of the antecedent credibility of an economic model can be broken down into more specific questions concerning the validity, comprehensiveness, robustness, reliability, and autonomy of the model. I will define these concepts in the following terms.

• Validity is a measure of the degree to which the assumptions employed in the construction of the model are thought to correspond to the real processes underlying the phenomena represented by the model.
• Comprehensiveness is the degree to which the model is thought to succeed in capturing the major causal factors that influence the features of the behavior of the system in which we are interested.
• Robustness is a measure of the degree to which the results of the model persist under small perturbations in the settings of parameters, formulation of equations, etc.
• Autonomy refers to the stability of the model's results in face of variation of contextual factors.
• Reliability is a measure of the degree of confidence we can have in the data employed in setting the values of the parameters.

These are epistemic features of models that can be investigated more or less independently and prior to examination of the empirical success or failure of the predictions of the model."

(Hausman's book is virtually definitive in its formulation of the tasks and scope of the philosophy of economics.  When conjoined with the book he wrote with Michael McPherson, Economic Analysis, Moral Philosophy and Public Policy, the philosophy of economics itself becomes a "separate science": virtually all the important questions are raised throughout a bounded domain, and a reasonable set of theories are offered to answer those questions.)

Read more »

Posted in: CAT_epistemology,confirmation,explanation,prediction

Predictions

Image: Artillery, 1911. Roger de La Fresnaye. Metropolitan Museum, New York

In general I'm skeptical about the ability of the social sciences to offer predictions about future social developments. (In this respect I follow some of the instincts of Oskar Morgenstern in On the Accuracy of Economic Observations.) We have a hard time answering questions like these:

• How much will the first installment of TARP improve the availability of credit within three months?
• Will the introduction of UN peacekeeping units reduce ethnic killings in the Congo?
• Will the introduction of small high schools improve student performance in Chicago?
• Will China develop towards more democratic political institutions in the next twenty years?
• Will American cities witness another round of race riots in the next twenty years?
  

However, the situation isn't entirely negative, and there certainly are some social situations for which we can offer predictions in at least a probabilistic form. Here are some examples:

• The unemployment rate in Michigan will exceed 10% sometime in the next six months.
• Coalition casualties in the Afghanistan war will be greater in 2009 than in 2008.
• Illinois Governor Blogojevich will leave office within six months.
• Germany will be the world leader in solar energy research by 2020 (link).
• The Chinese government will act strategically to prevent emergence of regional independent labor organizations.
  

It is worth exploring the logic and function of prediction for a few lines. Fundamentally, it seems that prediction is related to the effort to forecast the effects of interventions, the trajectory of existing trends, and the likely strategies of powerful social actors. We often want to know what will be the net effect of introducing X into the social environment. (For example, what effect on economic development would result from a region's succeeding in increasing the high school graduation rate from 50% to 75%?) We may find it useful to project into the future some social trends that can be observed in the present. (Demographers' prediction that the United States will be a "majority-minority" population by 2042 falls in this category (link).) And we can often do quite a bit of rigorous reasoning about the likely actions of leaders, policy makers, and other powerful actors given what we know about their objectives and their beliefs. (We can try to forecast the outcome of the current impasse between Russia and Ukraine over natural gas by analyzing the strategic interests of both sets of decision-makers and the constraints to which they must respond.)

So the question is, what kinds of predictions can we make in the social realm? And what circumstances limit our ability to predict?

Predictions about social phenomena are based on a couple of basic modes of reasoning:

• extrapolation of current trends
• modeling of causal hypotheses about social mechanisms and structures
  
• reasoning about strategic actions likely to be taken by actors
• derivation of future states of a system from a set of laws

And predictions can be presented in a range of levels of precision, specificity, and confidence:

• prediction of a single event or outcome: the selected social system will be in state X at time T.
• prediction of the range within which a variable will fall: the selected social variable will fall within a range Q ±20%.
  
• prediction of the range of outcome scenarios that are most likely: "Given current level of unrest, rebellion 60%, everyday resistance 30%, resolution 10%"
  
• prediction of the direction of change: the variable of interest will increase/decrease over the specified time period
  
• prediction of the distribution of properties over a group of events/outcomes. X percent of interventions will show improvement of variable Y.
  

Here are some particular obstacles to reliable predictions in the social realm:

• unquantifiable causal hypotheses -- "small schools improve student performance". How large is the effect? How does it weigh in relation to other possible causal factors?
  
• indeterminate interaction effects -- how will school policy changes interact with rising unemployment to jointly influence school attendance and performance?
  
• open causal fields. What other currently unrecognized causal factors are in play?
  
• the occurrence of unpredictable exogenous events or processes (outbreak of disease)
• ceteris paribus conditions. These are frequently unsatisfied.
  

So where does all this leave us with respect to social predictions? A few points seem relatively clear.

Specific prediction of singular events and outcomes seems particularly difficult: the collapse of the Soviet Union, China's decision to cross the Yalu River in the Korean War, or the onset of the Great Depression were all surprises to the experts.

Projection of stable trends into the near future seems most defensible -- though of course we can give many examples of discontinuities in previously stable trends. Projection of trends over medium- and long-term is more uncertain -- given the likelihood of intervening changes of structure, behavior, and environment that will alter the trends over the extended time.

Predictions of limited social outcomes, couched in terms of a range of possibilities attached to estimates of probabilities and based on analysis of known causal and strategic processes, also appear defensible. The degree of confidence we can have in such predictions is limited by the possibility of unrecognized intervening causes and processes.

The idea of forecasting the total state of a social system given information about the current state of the system and a set of laws of change is entirely indefensible. This is unattainable; societies are not systems of variables linked by precise laws of transition.

Read more »

Posted in: CAT_epistemology,causal mechanism,prediction,quantitative methods

Correspondence, abstraction, and realism

Science is generally concerned with two central semantic features of theories: truth of theoretical hypotheses and reliability of observational predictions. (Philosophers understand the concept of semantics as encompassing the relations between a sentence and the world: truth and reference. This understanding connects with the ordinary notion of semantics as meaning, in that the truth conditions of a sentence are thought to constitute the meaning of the sentence.) Truth involves a correspondence between hypothesis and the world; while predictions involve statements about the observable future behavior of a real system. Science is also concerned with epistemic values: warrant and justification. The warrant of a hypothesis is a measure of the degree to which available evidence permits us to conclude that the hypothesis is approximately true. A hypothesis may be true but unwarranted (that is, we may not have adequate evidence available to permit confidence in the truth of the hypothesis). Likewise, however, a hypothesis may be false but warranted (that is, available evidence may make the hypothesis highly credible, while it is in fact false). And every science possesses a set of standards of hypothesis evaluation on the basis of which practitioners assess the credibility of their theories--for example, testability, success in prediction, inter-theoretical support, simplicity, and the like.

The preceding suggests that there are several questions that arise in the assessment of scientific theories. First, we can ask whether a given hypothesis is a good approximation of the underlying social reality--that is, the approximate truth of the hypothesis. Likewise, we can ask whether the hypothesis gives rise to true predictions about the future behavior of the underlying social reality. Each of these questions falls on the side of the truth value of the hypothesis. Another set of questions concerns the warrant of the hypothesis: the strength of the evidence and theoretical grounds available to us on the basis of which we assign a degree of credibility to the hypothesis. Does available evidence give us reason to believe that the hypothesis is approximately true, and does available evidence give us reason to expect that the hypothesis's predictions are likely to be true? These questions are centrally epistemic; answers to them constitute the basis of our scientific confidence in the truth of the hypothesis and its predictions.

It is important to note that the question of the approximate truth of the hypothesis is separate from that of the approximate truth of its predictions. It is possible that the hypothesis is approximately true but its predictions are not. This might be the case because the ceteris paribus conditions are not satisfied, or because low precision of estimates for exogenous variables and parameters leads to indeterminate predictive consequences. Therefore it is possible that the warrant attaching to the approximate truth of the hypothesis and the reliability of its predictions may be different. It may be that we have good reason to believe that the hypothesis is a good approximation of the underlying economic reality, while at the same time we have little reason to rely on its predictions about the future behavior of the system. The warrant of the hypothesis is high on this account, while the warrant of its predictions is low.

Whatever position we arrive at concerning the possible truth or falsity of a given economic hypothesis, it is plain that this cannot be understood as literal descriptive truth. Economic hypotheses are not offered as full and detailed representations of the underlying economic reality. For a hypothesis unavoidably involves abstraction, in at least two ways. First, the hypothesis deliberately ignores some empirical characteristics and causal processes of the underlying economic reality. Just as a Newtonian hypothesis of the ballistics of projectiles ignores air resistance in order to focus on gravitational forces and the initial momentum of the projectile, so an economic hypothesis ignores differences in consumption behavior among members of functional defined income groups. Likewise, a hypothesis may abstract from regional or sectional differences in prices or wage rates within a national economy. Daniel Hausman provides an excellent discussion of the scope and limits of economic theories in The Inexact and Separate Science of Economics.

Another epistemically significant feature of social hypotheses is the difficulty of isolating causal factors in real social or economic systems. Hypotheses are generally subject to ceteris paribus conditions. Predictions and counterfactual assertions are advanced conditioned by the assumption that no other exogenous causal factors intervene; that is, the assertive content of the hypothesis is that the economic processes under analysis will unfold in the described manner absent intervening causal factors. But if there are intervening causal factors, then the overall behavior of the system may be indeterminate. In some cases it is possible to specify particularly salient interfering causal factors (e.g. political instability). But it is often necessary to incorporate open-ended ceteris paribus conditions as well.

Finally, social theories and hypotheses unavoidably make simplifying or idealizing assumptions about the populations, properties, and processes that they describe. Consumers are represented as possessing consistent and complete preference rankings; firms are represented as making optimizing choices of products and technologies; product markets are assumed to function perfectly; and so on.

Given, then, that hypotheses abstract from reality, in what sense does it make sense to ask whether a hypothesis is true? We must distinguish between truth and completeness, to start with. To say that a description of a system is true is not to say that it is a complete description. (A complete description provides a specification of the value of all state variables for the system--that is, all variables that have a causal role in the functioning of the system.) The fact that hypotheses are abstractive demonstrates only that they are incomplete, not that they are false. A description of a hockey puck's trajectory on the ice that assumes a frictionless surface is a true account of some of the causal factors at work: the Newtonian mechanics of the system. The assumption that the surface of the ice is frictionless is false; but in this particular system the overall behavior of the system (with friction) is sufficiently close to the abstract hypothesis (because frictional forces are small relative to other forces affecting the puck). In this case, then, we can say two things: first, the Newtonian hypothesis is exactly true as a description of the forces it directly represents, and second, it is approximately true as a description of the system as a whole (because the forces it ignores are small).

This account takes a strongly realist position on social theory, in that it characterizes truth in terms of correspondence to unobservable entities, processes, or properties. The presumption here is that social systems generally--and economic systems in particular--have objective unobservable characteristics which it is the task of social science theory to identify. The realist position is commonly challenged by some economists, however. Milton Friedman's famous argument for an instrumentalist interpretation of economic theory (Essays in Positive Economics) is highly unconvincing in this context. The instrumentalist position maintains that it is a mistake to understand theories as referring to real unobservable entities. Instead, theories are simply ways of systematizing observable characteristics of the phenomena under study; the only purpose of scientific theory is to serve as an instrument for prediction. Along these lines, Friedman argues that the realism of economic premises is irrelevant to the warrant of an economic theory; all that matters is the overall predictive success of the theory. The instrumentalist approach to the interpretation of economic theory, then, is highly unpersuasive as an interpretation of the epistemic standing of economic hypotheses. Instead, the realist position appears to be inescapable: we are forced to treat general equilibrium theory as a substantive empirical hypothesis about the real workings of competitive market systems, and our confidence in general equilibrium hypotheses is limited by our confidence in the approximate truth of the general equilibrium theory.

Read more »

Posted in: CAT_epistemology,epistemology,prediction,realism

Polling and social knowledge

Here's a pretty interesting graphic from Pollster.com:



As you can see, the graph summarizes a large number of individual polls measuring support for the two major party candidates from January 1 to October 26. The site indicates that it includes all publicly available polls during the time period. Each poll result is represented with two markers -- blue for Obama and red for McCain. The red and blue trend lines are "trend estimates" based on local regressions for the values of the corresponding measurements for a relatively short interval of time (the site doesn't explicitly say what the time interval is). So, for example, the trend estimate for August 1 appears to be approximately 47%:42% for the two candidates. As the site explains, 47% is not the average of poll results for Obama on August 1; instead, it is a regression result based on the trend of all of Obama's polling results for the previous several days.

There are a couple of things to observe about this graph and the underlying methodology. First, it's a version of the "wisdom of the crowd" idea, in that it arrives at an estimate based on a large number of less-reliable individual observations (the dozen or so polling results for the previous several days). Each of the individual poll results has an estimate-of-error which may be in the range of 3-5 percentage points; the hope is that the aggregate result has a higher degree of precision (a narrower error bar).

Second, the methodology attempts to incorporate an estimate of the direction and rate of movement of public opinion, by incorporating trend information based on the prior several days' polling results.

Third, it is evident that there is likely to be a range of degrees of credibility assigned to the various component polls; but the methodology doesn't assign greater weight to "more credible" polls. Ordinary readers might be inclined to assign greater weight to a Gallup poll or a CBS poll than a Research2000 or a DailyKos poll; but the methodology treats all results equally. Likewise, the critical reader might assign more credibility to a live phone-based poll than an internet-based or automated phone poll; but this version of the graph includes all polls. (On the website it is possible to filter out internet-generated or automated phone polling results; this doesn't seem to change the shape of the results noticeably.)

There is also a fundamental question of validity and reliability that the critical reader needs to ask: how valid and reliable are these estimates for a particular point in time? That is, how likely is it that the trend estimate of support for either candidate on a particular day is within a small range of error of the actual value? I assume there is some statistical method for estimating probable error for this methodology, though it doesn't appear to be explained on the website. But fundamentally, the question is whether we have a rational basis for drawing any of the inferences that the graph suggests -- for example, that Obama's lead over McCain is narrowing in the final 14 days of the race.

Finally, there is the narrative that we can extract from the graph, and it tells an interesting story. From January through March candidate Obama has a lead over candidate McCain; but of course both candidates are deeply engaged in their own primary campaigns. At the beginning of April the candidates are roughly tied at 45%. From April through September Obama rises slowly and maintains support at about 48%, while McCain falls in support until he reaches a low point of 43% in the beginning of August. Then the conventions take place in August and early September -- and McCain's numbers bump up to the point where Obama and McCain cross in the first week of September. McCain takes a brief lead in the trend estimates. His ticket seems to derive more benefit from his "convention bump" than Obama does. But in the early part of September the national financial crisis leaps to center stage and the two candidates fare very differently. Obama's support rises steeply and McCain's support falls at about the same rate, opening up a 7 percentage point gap in the trend estimates by the middle of October. From the middle of October the race begins to tighten; McCain's support picks up and Obama's begins to dip slightly at the end of October. But the election looms -- the trend estimates tell a story that's hard to read in any way but "too late, too little" for the McCain campaign.

And, of course, it will be fascinating to see where things stand a week from today.

Here is the explanation that the website offers of its methodology:

[quoting from Pollster.com:]

"Where do the numbers come from?

When you hold the mouse pointer over a state, you see a display of the latest "trend estimate" numbers from our charts of all available public polls for that race. The numbers for each candidate correspond to the most recent trend estimate -- that is the end point of the trend line that we draw for each candidate. If you click the state on the map, you will be taken to the page on Pollster.com that displays the chart and table of polls results for that race.

In most cases, the numbers are not an "average" but rather regression based trendlines. The specific methodology depends on the number of polls available.

• If we have at least 8 public polls, we fit a trend line to the dots represented by each poll using a "Loess" iterative locally weighted least squares regression.
  
• If we have between 4 and 7 polls, we fit a linear regression trend line (a straight line) to best fit the points.
  
• If we have 3 polls or fewer, we calculate a simple average of the available surveys.

How do regression trend lines differ from simple averages?

Charles Franklin, who created the statistical routines that plot our trend lines, provided the following explanation last year:

Our trend estimate is just that, an estimate of the trends and where the race stands as of the latest data available. It is NOT a simple average of recent polling but a "local regression" estimate of support as of the most recent poll. So if you are trying to [calculate] our trend estimates from just averaging the recent polls, you won't succeed.

Here is a way to think about this: suppose the last 5 polls in a race are 25, 27, 29, 31 and 33. Which is a better estimate of where the race stands today? 29 (the mean) or 33 (the local trend)? Since support has risen by 2 points in each successive poll, our estimator will say the trend is currently 33%, not the 29% the polls averaged over the past 2 or 3 weeks during which the last 5 polls were taken. Of course real data are more noisy than my example, so we have to fit the trend in a more complicated way than the example, but the logic is the same. Our trend estimates are local regression predictions, not simple averaging. If the data have been flat for a while, the trend and the mean will be quite close to each other. But if the polls are moving consistently either up or down, the trend estimate will be a better estimate of opinion as of today while the simple average will be an estimate of where the race was some 3 polls ago (for a 5 poll average-- longer ago as more polls are included in the average.) And that's why we estimate the trends the way we do."

Read more »

Posted in: CAT_methodology,epistemology,prediction,quantitative methods

Policy, treatment, and mechanism

Policies are selected in order to bring about some desired social outcome or to prevent an undesired one. Medical treatments are applied in order to cure a disease or to ameliorate its effects. In each case an intervention is performed in the belief that this intervention will causally interact with a larger system in such a way as to bring about the desired state. On the basis of a body of beliefs and theories, we judge that T in circumstances C will bring about O with some degree of likelihood. If we did not have such a belief, then there would be no rational basis for choosing to apply the treatment. "Try something, try anything" isn't exactly a rational basis for policy choice.

In other words, policies and treatments depend on the availability of bodies of knowledge about the causal structure of the domain we're interested in -- what sorts of factors cause or inhibit what sorts of outcomes. This means we need to have some knowledge of the mechanisms that are at work in this domain. And it also means that we need to have some degree of ability to predict some future states -- "If you give the patient an aspirin her fever will come down" or "If we inject $700 billion into the financial system the stock market will recover."

Predictions of this sort could be grounded in two different sorts of reasoning. They might be purely inductive: "Clinical studies demonstrate that administration of an aspirin has a 90% probability of reducing fever." Or they could be based on hypotheses about the mechanisms that are operative: "Fever is caused by C; aspirin reduces C in the bloodstream; therefore we should expect that aspirin reduces fever by reducing C." And ideally we would hope that both forms of reasoning are available -- causal expectations are born out by clinical evidence.

Implicitly this story assumes that the relevant causal systems are pretty simple -- that there are only a few causal pathways and that it is possible to isolate them through experimental studies. We can then insert our proposed interventions into the causal diagram and have reasonable confidence that we can anticipate their effects. The logic of clinical trials as a way of establishing efficacy depends on this assumption of causal simplicity and isolation.

But what if the domain we're concerned with isn't like that? Suppose instead that there are many causal factors and a high degree of causal interdependence among the factors. And suppose that we have only limited knowledge of the strength and form of these interdependencies. Is it possible to make rationally justified interventions within such a system?

This description comes pretty close to what are referred to as complex systems. And the most basic finding in the study of complex systems is the extreme difficulty of anticipating future system states. Small interventions or variations in boundary conditions produce massive variations in later system states. But this is bad news for policy makers who are hoping to "steer" a complex system towards a more desirable state. There are good analytical reasons for thinking that they will not be able to anticipate the nature or magnitude or even direction of the effects of the intervention.

The study of complex systems is a collection of areas of research in mathematics, economics, and biology that attempt to arrive at better ways of modeling and projecting the behavior of systems with these complex causal interdependencies. This is an exciting field of research at places like the Santa Fe Institute and the University of Michigan. One important tool that had been extensively developed is the theory of agent-based modeling -- essentially, the effort to derive system properties as the aggregate result of the activities of independent agents at the micro-level. And a fairly durable result has emerged: run a model of a complex system a thousand times and you will get a wide distribution of outcomes. This means that we need to think of complex systems as being highly contingent and path-dependent in their behavior. The effect of an intervention may be a wide distribution of future states.

So far the argument is located at a pretty high level of abstraction. Simple causal systems admit of intelligent policy intervention, whereas complex, chaotic systems may not. But the important question is more concrete: which kind of system are we facing when we consider social policy or disease? Are social systems and diseases examples of complex systems? Can social systems be sufficiently disaggregated into fairly durable subsystems that admit of discrete causal analysis and intelligent intervention? What about diseases such as solid tumors? Can we have confidence in interventions such as chemotherapy? And, in both realms, can the findings of complexity theory be helpful by providing mathematical means for working out the system effects of various possible interventions?

Read more »

Posted in: CAT_mechanisms,causal reasoning,complexity,mechanism,prediction

System tendencies?

A central theme of many of the posts here is the contingency, heterogeneity, and path dependency of social processes. I used the metaphor of a "constrained random walk" in an earlier posting to characterize many social processes. This figure is intended to stand in contrast to the idea of an inevitable development towards an optimum or equilibrium point, on the one hand, or the idea of an inevitable system failure, on the other.

The idea here is this: from starting point A, there are numerous possible states of affairs Oi that might be reached over an extended period of time. There is no sense in which the course from A to the actual historical outcome Om is inevitable or unique. (From the starting point of Europe in 1910, including the social, political, and economic realities of the nations of Europe, multiple outcomes were accessible by the time of 1920: exhausting war, emergence of new and effective international organizations that sustained the peace, inspired just-in-time diplomacy bringing hostilities to an early termination, ...). Each of the pathways leading from A to Oi might be individually explicable, in terms of the situations of structure and agency that were present during the period of development. Virtually every point in the "space" of outcomes would be accessible, although some outcomes might be substantially less likely than others. Along the way there are likely to be cul-de-sacs; but in the aggregate, the space of possible outcomes from many historical starting points covers the full sphere of possibilities. Putting the point crudely, you can get anywhere from anywhere.

This conception emphasizes deep contingency in social change. But what about the symmetrical facts of "constraint" and "imperative" -- the limitations imposed by existing institutions and organizations at any specific stage and the positive impulses to change that are often embodied in the incentive structures of existing institutions? Is the contingency of social events to some extent reduced by the relative durability of existing core social institutions? Is there such a thing as a "logic of institutions" that is embodied in a particular configuration of core social institutions, with the result that societies embodying these institutions will be most likely to develop in one way rather than another?

This description lies at the heart of Marx's analysis of social systems as modes of production. Marx believed that the core institutions that defined the property system, the system of labor control, and the distribution of wealth have deep effects on individual agency, leading and constraining agents to behave in ways that lead in the aggregate to certain kinds of social outcomes. Modes of production have system tendencies that can be inferred from their basic institutional features. A particularly clear example is his analysis of the "law" of the falling rate of profit within capitalism: firms are required to maximize profits; they have the opportunity of introducing capital-intensive technologies that lower costs, thereby increasing profits in the short run; competition with other profit-maximizing firms pushes prices down to the new cost of production; the rising capital-labor ratio in industry creates a falling rate of profit. So capitalism embodies a system tendency towards a falling rate of profit over time. Similar reasoning underlies Marx's prediction of financial crises within capitalism. (See an earlier posting on Marx's conception of capitalism.)

And in fact, if we could make two assumptions, then Marx's reasoning about the tendencies of capitalism would be very compelling: the assumption that the core economic institutions are fixed and unchanging, and the assumption that there are no other social-political-economic institutions in play that might serve as resources for policies and actions that would offset the predicted tendencies of capitalism. However, neither of these assumptions is correct. The institutions of any major social order -- feudalism, the Chinese agrarian economy, capitalism, state socialism -- are always the composite of a vast number of lower-level institutions; and these lower-level institutions are usually in a state of flux. So the core institutions are not fixed and unchanging. The traditional Chinese agrarian economy was remarkably resilient in face of a range of deep challenges over centuries; adjustment of basic social institutions permitted Chinese society to cope better with environmental and international circumstances than a modeled Chinese economy would have predicted.

Second, even more fundamentally, a society is not simply a "mode of production," constituted by an economic structure. Rather, there are a range of other, equally fundamental institutions and practices -- cultural, political, legal, community-based and national -- through which resourceful agents attempt to solve personal or social problems at various points in time. So the "logic" of the economic institutions is only one part of the overall social trajectory; instead, we have the strategic interaction and aggregation of political, cultural, social, demographic, and legal institutions that complement and offset the workings of the economic structure. And further, we can correctly say that each of these aspects of social organization has its own "system tendencies." Elected legislatures have a logic that derives from the calculations of political self-interest of the legislators, community-based organizations have their own logic, various demographic regimes have their own tendencies (for example, the favoring of boy children produces skewed sex ratios that have negative political effects), and so forth.

So the tentative conclusion that I draw from these various considerations is, once again, to give the nod to contingency while recognizing the partial imperatives created by the various sets of core institutions that are embodied in a society at a given time. Structures do of course constrain agents. But structures interact with each other, leading to surprising results. And structures change in response to a variety of causes, including the strategic efforts of agents to modify them. So the upshot is, once again, that we should expect a high degree of contingency in outcomes over extended periods of historical time. Historical experience may well support the discovery that "capitalism creates a tendency towards X" or "fascist politics create a tendency towards Y". To that extent, there are "system tendencies". But it is rare for one particular sub-system (property relations, electoral system, demographic regime) to dominate the overall historical trajectory. And so the system tendencies of one partial set of core institutions rarely become the system tendencies of the overall social whole.

Read more »

Posted in: CAT_explanation,history,Marx,prediction,structure

What social science can do

Quite a few postings here emphasize the limits of social science knowledge. Prediction of the behavior of large social wholes is difficult to impossible. There are few strong regularities among social phenomena. Social entities and processes are heterogeneous, plastic, and path-dependent. So the question arises: what can the social sciences do that takes them beyond the realm of description and reportage of the blooming, buzzing confusion of social comings and goings, to something that is more explanatory and generalizable?

I think there is an answer to this, and it has to do with identifying mid-level mechanisms and processes that recur in roughly similar ways in a range of different social settings. The social sciences can identify a fairly large number of these sorts of recurring mechanisms. For example --

• public goods problems
• political entrepreneurship
• principal-agent problems
• features of ethnic or religious group mobilization
• market mechanisms and failures
• rent-seeking behavior
• the social psychology associated with small groups
• the moral emotions of family and kinship
• the dynamics of a transport network
• the communications characteristics of medium-size social networks
• the psychology and circumstances of solidarity

Further, the social sciences can attempt to discover the circumstances at the level of individual agents that make these mechanisms robust across social settings. They can model the dynamics and features of aggregation that they possess. And they can attempt to discover the workings of such mechanisms in particular social and historical settings, and work towards explanations of particular features of these events based on their theories of the properties of the mechanisms. Finally, they can attempt to find rigorous ways of attempting to model the effects of aggregating multiple mechanisms in a particular setting.

What this comes down to is the view that the main theoretical and generalizing contribution that the social sciences can make is the discovery and analysis of a wise range of recurring social mechanisms grounded in features of human agency and common institutional and material settings. They can help to constitute a rich tool box for social explanation. And, in a weak and fallible way, they can lay the basis for some limited social generalizations -- for example, "In circumstances where a group of independent individuals make private decisions about their actions, the public goods shared by the group will be under-provided."

This approach affords a degree of explanatory capacity and generalization to the social sciences. What it does not underwrite is the ability to offer general, comprehensive theories about any complex kind of phenomenon -- cities, schools, revolutions. And it does not provide a foundation for confidence about large predictions about the future behavior of complex social wholes.

Read more »

Posted in: CAT_explanation,causal mechanism,explanation,prediction

Social surprises

The near meltdown of the US financial system this week came as a surprise to most of us -- experts, legislators, and citizens alike. That isn't to say that the components of the disaster were unknown -- the subprime crisis, the earlier financial undoings of Fannie Mae and Bear Stearns this summer, and the sudden collapse of Lehman Brothers last week. But what has come as a surprise is the severity of the warnings by the Federal Reserve and Treasury that the entire financial system is only a few steps from seizure and collapse. This is a catastrophic system failure -- and no one would have anticipated its possibility six months ago.

Think of a few other surprises in the past thirty years -- the collapse of the Soviet Union, the Iranian Revolution, or the emergence of China as a roaring engine of market-based growth. In each case the event was a discontinuous break from the trajectory of the past, and it surprised experts and citizens alike. (The photo above depicts the surprising Yeltsin standing on a tank in 1991.)

So what is a surprise? It is an event that shouldn't have happened, given our best understanding of how things work. It is an event that deviates widely from our most informed expectations, given our best beliefs about the causal environment in which it takes place. A surprise is a deviation between our expectations about the world's behavior, and the events that actually take place. Many of our expectations are based on the idea of continuity: tomorrow will be pretty similar to today; a delta change in the background will create at most an epsilon change in the outcome. A surprise is a circumstance that appears to represent a discontinuity in a historical series.

It would be a major surprise if the sun suddenly stopped shining, because we understand the physics of fusion that sustains the sun's energy production. It would be a major surprise to discover a population of animals in which acquired traits are passed across generations, given our understanding of the mechanisms of evolution. And it would be a major surprise if a presidential election were decided by a unanimous vote for one candidate, given our understanding of how the voting process works. The natural world doesn't present us with a large number of surprises; but history and social life are full of them.

The occurrence of major surprises in history and social life is an important reminder that our understanding of the complex processes that are underway in the social world is radically incomplete and inexact. We cannot fully anticipate the behavior of the subsystems that we study -- financial systems, political regimes, ensembles of collective behavior -- and we especially cannot fully anticipate the interactions that arise when processes and systems intersect. Often we cannot even offer reliable approximations of what the effects are likely to be of a given intervention. This has a major implication: we need to be very modest in the predictions we make about the social world, and we need to be cautious about the efforts at social engineering that we engage in. The likelihood of unforeseen and uncalculated consequences is great.

And in fact commentators are now raising exactly these concerns about the 700 billion dollar rescue plan currently being designed by the Bush administration to save the financial system. "Will it work?" is the headline; "What unforeseen consequences will it produce?" is the subtext; and "Who will benefit?" is the natural followup question.

It is difficult to reconcile this caution about the limits of our rational expectations about the future based on social science knowledge, with the need for action and policy change in times of crisis. If we cannot rely on our expectations about what effects an intervention is likely to have, then we can't have confidence in the actions and policies that we choose. And yet we must act; if war is looming, if famine is breaking out, if the banking system is teetering, a government needs to adopt policies that are well designed to minimize the bad consequences. It is necessary to make decisions about action that are based on incomplete information and insufficient theory. So it is a major challenge for the theory of public policy, to attempt to incorporate the limits of knowledge about consequences into the design of a policy process. One approach that might be taken is the model of designing for "soft landings" -- designing strategies that are likely to do the least harm if they function differently than expected. Another is to emulate a strategy that safety engineers employ when designing complex, dangerous systems: to attempt to de-link the subsystems to the extent possible, in order to minimize the likelihood of unforeseeable interactions. (Nancy Leveson describes some of these strategies in Safeware: System Safety and Computers.) And there are probably other heuristics that could be imagined as well.

Read more »

Posted in: CAT_explanation,heterogeneity,policy,prediction

What is social scientific knowledge?

The social and behavioral sciences endeavor to describe, explain, and interpret the range of the social and behavioral facts that surround us. To refer to this body of findings as “science” is to claim a set of epistemic values about the nature of the methods of inquiry and evaluation that are used to arrive at and assess the conclusions offered about this domain. The label “science” brings with it a set of presuppositions about rigor, evidence, generalizability, logical analysis, objectivity, cumulativeness, and the likelihood that the assertions that are made are true.

Consider a few assumptions that are often made about scientific knowledge—some valid and some not. Science is based on a set of rationally justified methods of inquiry and testing. Scientific knowledge progresses, in scope, in detail of understanding, and in reliability. Science is performed by specialists, working within equally exacting communities of peers and competitors and subject to a demanding set of standards of evaluation—peer-reviewed journals, university review processes, national laboratories, and international associations and conferences. The result of these processes of testing and evaluation, we expect, is an expanding body of hypotheses, experimental findings, observations, theories, and explanations that have substantial credibility—and substantially higher credibility than the writings of casual observers of a given range of phenomena. We come to know the nature of the world better through the institutions and methods of science.

In addition to these reasonably valid assumptions about scientific knowledge, there is another group of more questionable ideas that derive from assumptions drawn from the natural sciences. Science permits generalizations; it permits us to systematize otherwise apparently separate domains of phenomena (planetary motion, the tides; rational choice theory, behavior of the family) and to demonstrate that apparently heterogeneous sets of phenomena are in fact governed by the same general laws. Science permits predictions; if the fundamentals are thus-and-so, then the compounds will behave thusly. Science aims at unification: the discovery of unitary systems of forces and entities whose aggregate properties represent the whole of nature.

Notice that these latter expectations are derived from the successes and specific characteristics of certain of the natural sciences. And this marks the first of many opportunities for error in the philosophy of social science. There is no reason to expect that the social domain possesses the underlying nature and orderliness that would make it possible to achieve some of these characteristics (in particular, uniformity, generalizability, unification, simplicity). Consider some other areas of possible empirical research—for example, animal behavior. We should not expect there to be comprehensive theories of animal behavior. Instead, we should expect many threads of research, corresponding to many dimensions of animal behavior: cognition, memory, instinct, social behaviors, migratory behavior. And these many strands of research would reach out to different kinds of causal backgrounds: evolutionary biology, neurophysiology, intra-group learning. Likewise with the domain of social behavior. There is no single unified “theory of human motivation”—whether rational choice theory, social psychology, or any other unified theory. And this is so, because there is no unified reality of motivation and action; rather, there is a heterogeneous range of motives, errors, impulses, commitments, and habits that together constitute “dispositions to behave.”

What underwrites the claim of truthfulness for scientific knowledge? What gives us a rational basis for believing that the results of the socially constructed activities of science lead to true hypotheses about the nature and workings of the phenomena that scientific inquiry considers? There is, first, the basic argument of empiricism: we can observe some features of the world and establish certain statements as being probable. And we can use a collection of tools of inference to establish credibility of other non-observational statements (deductive and inductive logic, statistics, the experimental method, causal modeling).

This simple empiricist epistemology underwrites the strongest claims for veridicality and justification for the social sciences. The discovery of empirical facts about the social world is possible but challenging; this is what much of social science methodology attempts to under-gird. And hypotheses about the causal relationships that exist among social entities and processes can be tested using a variety of methods of inference that themselves possess strong epistemic justification. We have learned from the writings of philosophers of science since the 1960s to emphasize corrigibility and anti-foundationalism in our interpretation of scientific knowledge; but a coherentist epistemology and a perspective of causal realism provides a philosophically powerful grounding for social science knowledge. (See articles in the Stanford Encyclopedia of Philosophy on coherence epistemology and scientific realism by Kvanvig and Boyd).

In addition, in some areas of the natural sciences, there is the fact that cumulative scientific research leads to the invention of technologies that work as they were designed to do: new materials are invented in the electronics industry, new designs are created for large structures (buildings, aircraft, electron microscopes), and these materials and artifacts perform as expected on the basis of the underlying theories. So scientific theories of materials, structures, and natural systems are supported by the effectiveness of the technologies that they give rise to. If the theories and hypotheses were fundamentally untrue about the parts of the natural world that they describe, then we would expect the technologies to fail; the technologies do not fail; so we have some additional reason to believe the scientific theories that underlay these technologies. (This is a version of Richard Boyd’s argument of methodological realism.)

Is there anything analogous to the relationship between the natural sciences and technology, for the social and behavioral sciences? On the whole, there is not. Social predictions are notoriously unreliable; public policies based on social-science theory commonly give rise to unanticipated consequences; and the twists and turns of deliberate social processes (war, alliance, efforts to address global warming) continue to surprise us. This unpredictability in the social world derives from the nature of social action. Human behavior and social processes are plainly subject to an open-ended range of causes, motives, and influences. The construction of various areas of science with the goal of understanding and explaining this multiplicity is therefore a profoundly challenging task.

Here, then, is a very elliptical description of a plausible interpretation of social science epistemology: There are empirical foundations for knowledge in the form of social observation (empiricism); there are social causes that influence social behavior, processes, and outcomes (causal realism); there is no a priori reason to expect strong generalizations across social phenomena, “regulating” the social world; and there is no reason to expect unified master theories of social phenomena, suggesting instead a preference for theories of the middle range.

Read more »

Posted in: CAT_epistemology,epistemology,philosophy of science,prediction