John Rawls's A Theory of Justice was a strikingly original contribution to political philosophy upon its appearance in 1971. Against the prevailing preference for "meta-ethics" in the field of philosophical ethics, Rawls made an effort to arrive at substantive, non-tautological principles that could be justified as a sort of "moral constitution" for a just society. The theory involves two fundamental principles of justice: the liberty principle, guaranteeing maximal equal liberties for all citizens, and the difference principle, requiring that social and economic inequalities should be the least possible, subject to the constraint of maximizing the position of the least-well-off. (The principle also requires equality of opportunity for all positions.)
Two elements of Rawls's philosophical argument were particularly striking. The first was his adoption of the anti-foundationalist coherence epistemology associated with Quine and Goodman (SEP article by Jonathan Kvanvig); so Rawls conceded that it is not possible to provide logically decisive arguments for moral positions. Though his theory of justice has much in common with the ideas of Kant and Rousseau, Rawls rejected the Kantian idea that moral theories could be given secure philosophical foundation. It is rather a question of the overall fit between a set of principles and our "considered judgments" about cases and mid-level moral judgments. He refers to the situation of "reflective equilibrium" as the state of affairs that results when a moral reasoner has fully deliberated about his/her considered moral judgments and tentative moral principles, adjusting both until no further changes are required by the requirement of consistency.
Another and perhaps even more distinctive part of Rawls's approach is his use of the apparatus of decision theory to support his arguments in favor of the two principles of justice against plausible alternatives (including especially utilitarianism). Essentially the argument goes along these lines. Suppose that representative individuals are brought together in a situation in which they are expected to make a unanimous and irreversible decision about the fundamental principles of justice that will regulate their society; and suppose they are profoundly ignorant about their own particular characteristics. Participants do not know whether they are talented, strong, intelligent, or eloquent; and they do not know what their fundamental goals are (their theories of the good). Rawls refers to this situation of choice as the original position; and he refers to the participants as deliberating behind the veil of ignorance. Rawls argues that rational individuals in these circumstances would unanimously choose the two principles of justice over utilitarianism. And this conclusion is taken to be a strong basis of support for the two principles as correct. This is what qualifies Rawls's theory as falling within the social contract tradition; the foundation of justice is the fact of unanimous rational consent (albeit hypothetical).
Once we connect the question, "what is the best theory of justice?", with the question, "what principles of justice would rationally self-interested persons choose?", there are various ways we might proceed. Rawls's description of the original position is just one possible starting point out of several. But if we begin with Rawls's assumptions, then it is natural to turn to formal decision theory as a basis for answering the question. How should rational agents reason in these circumstances? How should they decide which of several options will best serve their future interests? And one point becomes clear immediately: the choice of a decision rule makes a critical difference for the ultimate choice. If we were to imagine that decision-making under conditions of uncertainty mandates the "maximize expected utility" rule, then one choice follows (utilitarianism). But Rawls argues that the expected utility rule is not rational in the circumstances of the original position. The stakes are too high for each participant. And therefore he argues that the "maximin" rule would be chosen by rational participants in the circumstances of the original position. The maximin rule requires that we rank options by their worst possible outcome; and we choose that option that comes with the least bad outcome. In other words, we "maximize the minimum." (The maximin rule was described by von Neumann and Morgenstern in 1944 in their Theory of Games and Economic Behavior.)
Notice that this analysis involves a question of second-order rationality: not "what outcome would the rational agent choose?", but rather "what decision rule would the rational agent follow?". So it is the rationality of the decision rule rather than the rationality of the choice that is at issue.
Another important qualification has to do with defining more carefully what part of the theory of rationality Rawls is using in this argument. It is sometimes said that Rawls applies game theory to the situation of the original position; and there is a certain logic to this interpretation. Game theory is the theory of strategic rationality; it pertains to that set of situations in which the payoff for one participant depends on the rational choices of other participants. And the original position seems to embody this condition. However, the requirement of unanimity and the complete absence of a context of bargaining makes the situation non-strategic. So Rawls's use of rational choice theory does not involve game theory per se, and he is not interested in demonstrating a Nash equilibrium in the OP. Instead, he believes that there is a single best strategy that will be chosen by each individual--the two principles of justice. (Here is a good brief description of the main assumptions of game theory.)
One might ask whether the two features singled out here -- anti-foundationalism and decision theory -- are consistent. If Rawls's theory of justice depends on an argument within formal decision theory, then why is it not a foundationalist argument? (And in fact, Rawls on occasion refers to his argument as reflecting a "kind of moral geometry".) What makes Rawls's use of decision theory "anti-foundationalist" is the fact that this argument itself is philosophically contestable. Reasonable decision theorists may differ about the rationality of the maximin rule (as John Harsanyi argued against Rawls). So the appeal to decision theory does not obviate the need for a balance of reasons in favor of the approach and the particular way in which it is specified in this situation; and this in turn sounds a lot like the role of physical theory and methodology within Quine's notion of "The Web of Belief."
(A mountain of words have been written about Rawls's moral epistemology. Here is Samuel Freeman's excellent article on the original position in the Stanford Encyclopedia of Philosophy; here is a useful compendium of the history of rational choice theory; and here is an old article of mine on the epistemology of reflective equilibrium.)