Race has been a fundamental fact in American society for centuries, since the sixteenth century with the arrival of African slaves. And many would observe that racism has been a part of that history from beginning to end. These are distinct statements; it is possible for race to be a factor, without racism being present. But our history does not suggest this separation. Instead, the United States has embodied a pretty deep version of racial awareness, extending back to the period of slavery and its aftermath, and it continues to embody behaviors, attitudes, and outcomes that are best described as racist.
So what do we mean when we say that a society contains a substrate of racism? Can we observe and measure the social dimensions of racism? And can we say with any confidence that there has been change over time? Most fundamentally, a society is racist if members of one racial group despise, demean, mistreat, and discriminate against members of another racial group. Assertions of racial superiority and inferiority, patterns of treatment that discriminate across individuals within different racial groups, and outcomes that show a distinct advantage to members of a dominant racial group all represent the markings of racism within a society.
We might say that there are several important social factors that represent different aspects of the social reality of racism: attitudes, behaviors, institutions, and outcomes.
The attitudes of racism include a bundle of emotions and beliefs: a belief in the inherent superiority or inferiority of one race relative to another; feelings of hostility, suspicion, or antipathy towards members of a different racial group; a set of stereotypes about the characteristics of the other group; and a readiness to discriminate against members of other groups when one is in a position to assign benefits, opportunities, or hardships. There is such a thing as an "ideology" of race: a set of beliefs about people and the world that validates the assumptions of superiority and inferiority and the situation of privilege of the dominant group. It is possible to use survey methodology, focus groups, and individual interviews to probe attitudes of a population, and perhaps it is possible to "map" the variations in racial thinking throughout a society. So we might have some confidence in the possibility of developing an aggregate measure of "degree of racist attitudes" in a given society, and with effort we could monitor the direction of change of this measure over time. This might give us a basis for concluding that "racial attitudes are improving (or worsening) during a specific period."
Behaviors have something to do with attitudes; we generally believe that people's behaviors derive in some way from their underlying attitudes and ideologies. But the evidence of racist behavior is more visible than inferences about attitudes. Particularly visible are facts about the incidence of racially motivated violence; facts about discrimination in employment and housing; and patterns of social behaviors in interactions between members of different racial groups. A situation in which members of a privileged group express dominance, superiority, and greater self-importance towards members of a different group is one that expresses behaviorally the logic of race relations in that society. So we can ask the question of a given society: to what extent do members of a privileged racial group engage in harmful and discriminatory behaviors towards members of another group? And we might attempt to estimate the degree of racism in a society by the relative frequency of racially motivated crimes over time (FBI hate crime statistics; link). Consider this graph of the frequency of lynchings in the United States between 1865 and 1965 (Wikipedia link):
We might speculate that decades in which there is an exceptionally high incidence of lynchings are also peak periods of racism more generally. (Though we might also explain the frequency of these acts of violence in more political-organizational terms.)
The third factor mentioned above is institutions: to what extent are the basic institutions in a given society "racialized"? That is, to what extent do the basic institutions automatically and systematically treat members of different races differently? The practices of real estate "steering" and mortgage redlining are clear examples of a racialized system for assigning home seekers to neighborhoods; and employment practices that disadvantage members of some racial groups are just as critical in the area of employment and wellbeing. For example, facts about residential segregation and the availability of transportation make it extremely difficult for African-American applicants to pursue job openings in the suburbs. So this is a system that discriminates against one group in favor of another -- even though the employer's personal attitudes may not be the cause of the discrimination. Here too it is plausible to imagine that we might arrive at some quantifiable judgments about the degree of discrimination and equality of basic institutions at a given time; and this would permit us to make comparisons over time as well.
The fourth factor is outcomes. The degree of racialization of a society might be measured by the breadth of the gap between races with respect to important life outcomes. If blacks and whites differ significantly in life expectancy, incarceration rates, health status, income, wealth, and education, we have good reason to believe that there are racialized social processes that are leading to these outcomes. So measuring the race gap with respect to important social outcomes is an important way of assessing the degree to which a given society is racialized; or in other words, to measure the degree to which racism is an important social factor in that society. Racial gaps with respect to important life outcomes can be measured; and it is meaningful to find that racial gaps have narrowed (or widened) during a given period of time.
The fundamental test of a non-racist society is this: there should be no difference in the availability of opportunities across racial groups, and -- given a reasonable assumption about the fundamental equality of human beings -- there should be no gap in outcomes across racial groups when it comes to factors like health, education, income, or wealth. In other words, a non-racist society is one whose basic institutions do not discriminate, consciously or unconsciously, across individuals from different racial groups.
The treatment of attitudes, behaviors, institutions, and outcomes offered here suggests that it is indeed possible to chart the degree of racial progress a society has made over a number of decades. We can measure attitudes and behaviors over time, using rigorous social-science tools (survey methodology). And we can assess the workings of institutions and the distribution of outcomes using the suite of tools available to the social historian more generally. What is genuinely surprising, however, is the relative paucity of social-science research on this topic.
There is of course the crucial question of the social dynamics of racist attitudes and behavior. Are there social or psychological factors that make the appearance of racist attitudes more likely? Are there features of human nature that lead naturally to a social psychology of "in-group, out-group" discrimination? Or is racism a purely contingent accident that results from some important historical events in the past -- the slave economy, for example? What are the factors in contemporary American social life that are either conducive or inhibiting to the formation of racist attitudes and behaviors in individuals?