Procedural rationality, endogenous sampling, and the instability of defection

GAICIL Insights

Most attempts to rescue cooperation fiddle with the payoffs: fine the defector, reward the cooperator.¹ Procedural rationality, endogenous sampling, and the instability of defection, by Srinivas Arigapudi and Jonathan Newton, points to a different dial altogether: how people learn. The authors study populations whose members choose strategies through procedural rationality, testing strategies directly, and through imitation, observing others, and ask when universal defection in the prisoner's dilemma stops being stable.

The paper introduces a broad class of sampling-based decision rules covering both procedural rationality and imitation, studies population dynamics under them for prisoner's dilemmas, and characterizes necessary and sufficient conditions under which universal defection is unstable. Stability depends on the procedure by which strategies are sampled, the amounts of direct and indirect information agents use, and the game's payoff parameters.

The headline is stated plainly: for cooperation to emerge and persist, the manner in which agents generate and share information can be as important as the underlying payoff incentives. Interventions that alter learning procedures, alongside incentives, may shift long-run outcomes. A society's epistemology, its habits of testing and copying, becomes an institution in its own right.

Where does this leave AI? Artificial intelligence makes this result concrete. AI systems increasingly interact in markets, from algorithmic pricing to automated negotiation and trading, and they learn through procedures that blend direct experimentation with observation of other agents. On this framework, the learning procedure coded into an AI agent helps decide whether cooperative or defecting outcomes take hold. A line in a training loop becomes an institution.

That cuts both ways, and the paper's value is that it equips both sides. Sometimes cooperation among AI agents is valuable: coordinating on standards, sharing resources efficiently, avoiding destructive conflict. Learning procedures that destabilize universal defection could support it. Elsewhere, as with sellers' pricing in competitive markets, cooperation would hurt consumers. There, the same insights help spot learning procedures that might inadvertently sustain collusive outcomes. Designers and regulators read the same theorem for opposite purposes.

Humans are learning differently too. People increasingly observe others' strategies through digital platforms where AI curates what they see. The amount and type of indirect information people receive about others' choices is algorithmically shaped. If the mix of direct and indirect information decides whether cooperation persists, platform design belongs to the institutional environment of cooperation. A feed that shows people what their neighbors chose is, in this model, a cooperation policy.

The policy implication widens the toolkit. Taxes, subsidies and penalties change payoffs. This research shows that changing how people sample and share information can also move outcomes. Making the results of cooperative strategies visible, or encouraging direct experimentation, may destabilize defection at lower cost than reengineering incentives. Information can be cheaper than money.

The authors mark the next frontier: games beyond the prisoner's dilemma. Coordination games, public goods games and bargaining games all arise in AI-mediated environments, and extending the analysis would clarify how learning procedures shape a broad range of strategic interactions.

Stepping back, cooperation depends on learning as well as incentives. Populations that test strategies and observe others in particular proportions can escape universal defection. As AI agents join economic and social interactions, and as AI shapes how humans learn from one another, the design of learning procedures becomes a powerful lever. This paper supplies the theory for using that lever to promote cooperation where it creates value and to guard against it where it harms competition.

One question to put to AI governance: audits of algorithmic agents should inspect how they learn, alongside what they are paid to optimize. Two agents with identical objectives and different sampling habits may lead markets to very different places.