A person receiving unemployment payments in most wealthy democracies is required to demonstrate, at regular intervals, that they are actively seeking work. The demonstration requires a log of job search activities: applications made, positions considered, agencies contacted, training attended.
The log must be maintained in a specified format, submitted by a specified deadline, and verified by a specified authority. The activities that populate the log must meet criteria defined by the system: the applications must be to positions the system considers appropriate, the agencies must be registered with the relevant authority, the training must be approved by the relevant programme. The person whose job search does not conform to these specifications risks a reduction or suspension of the payment.
The time required to maintain the log, attend the appointments, and navigate the portal through which all of this is administered is not trivial. For a person who is also managing the practical consequences of unemployment—the financial stress, the disruption to daily structure, the task of finding work in a labour market that may not readily accommodate their skills or location—the administrative load of the payment’s conditions is a significant additional demand on time and cognitive resources that would otherwise be available for the job search the conditions are designed to support.
This is the compliance trap: the system’s conditions for receiving support consume a portion of the resources the support was supposed to free up for addressing the underlying situation. The trap is not designed. It is the predictable output of an incentive architecture that rewards visible compliance monitoring over effective outcome production, combined with an administrative apparatus whose design was optimised for processing cases rather than for the experience of the person being processed.
The full-time nature of navigating a complex welfare system is not a metaphor. Research across multiple jurisdictions has documented the hours per week that welfare recipients spend on system-related activities: the phone calls that do not connect, or connect to a queue, or connect to a person who cannot resolve the issue and must transfer the call; the portal sessions that time out before the form is completed; the appointments that require transport the person cannot easily afford; the documentation requests for records that must be obtained from agencies that have their own access requirements and waiting periods; the appeals against decisions that require written submissions and scheduled hearings. Each of these is a legitimate administrative activity from the system’s perspective. Their accumulation, for the person who must navigate all of them simultaneously, is a workload.
The workload is heavier for the people who are least equipped to bear it. The person with a mental health condition that impairs concentration and executive function is navigating a system whose compliance requirements are extensive and whose tolerance for non-compliance is low.
The person with a physical disability that limits mobility is navigating a system whose offices and appointment requirements often assume mobility.
The person whose first language is not the language of the system’s forms and portals is navigating a system whose communications assume a level of linguistic and bureaucratic fluency that the system has not provided resources to develop. The compliance load falls most heavily on the people whose capacity to bear it is most limited by the conditions that produced their need for support.
The system does not know this about itself in any systematic way. It records compliance and non-compliance. It does not record the effort that compliance required, or the cost of that effort in terms of the resources it diverted from the underlying situation the compliance was supposed to address.
The professionalisation of care is a development that followed directly from the recognition that informal community support—the mutual aid networks, the neighbourhood connections, the family and friendship structures through which people have always supported each other through difficulty—was insufficient to address the scale and complexity of need in industrialised societies.
The recognition was accurate. The informal networks were insufficient. They were also unequally distributed, leaving people without family, without community, without the social capital that mutual aid requires, without support of any kind. The professional care system was a genuine improvement on the alternative of no system.
The improvement came with a cost that was not initially visible because the comparison was with inadequacy. The informal support that the professional system supplemented or replaced had qualities that the professional system could not replicate at scale.
The neighbour who helped because they knew the person and cared about them was operating within a relationship of ongoing mutual obligation. The professional who helped because it was their job was operating within a relationship defined by role, contract, and the accountability structures of the employing institution. Both forms of help can be genuine. They are not identical in their effects on the person receiving them, or in their effects on the social fabric within which the help occurs.
The informal network, when it functions, is self-sustaining: the person who receives help is also a person who gives it, and the reciprocity maintains the network.
The professional system is sustained by funding, which requires the demonstration of need, which requires assessment, which requires documentation, which returns us to the compliance architecture. The person who is integrated into a professional care system and removed from informal networks—because the professional system requires them to formalise needs that were previously managed informally, or because the professionalisation signals that informal support is unnecessary, or because the time required for professional system compliance leaves less time for the informal relationships through which mutual support occurs—may receive more documented support while experiencing less genuine connection.
The organic community network, where it existed, did not require a referral. It did not have opening hours. It did not produce a case record. It produced presence—the irregular, unscheduled, genuinely responsive presence of people who noticed when something was wrong and responded to what they noticed.
The professional system produces appointments. The appointments are real. The presence that the organic network once provided, in the gaps between the appointments and outside the categories the appointments address, is the thing the professional system has the most difficulty replicating.
The automated decision system is the point at which the epistemic degradation of welfare administration becomes most acute and most difficult to correct. The human administrator who makes an incorrect decision can be asked to reconsider. The administrator can receive new information, assess its relevance, and revise the decision. The administrator’s decision-making can be challenged through appeal mechanisms designed to apply human judgment to disputed determinations. The automated system’s decision is the output of an algorithm whose parameters were set by people who are not present in the individual case and whose reasoning cannot be inspected by the person affected by it.
The opacity is not always intentional.
The algorithms that determine welfare eligibility, flag accounts for fraud investigation, calculate benefit amounts, and generate compliance requirements are often genuinely complex, and the complexity that makes them capable of processing large numbers of cases consistently is also the complexity that makes their reasoning opaque to the people whose situations they are processing. The person who receives an automated determination and believes it to be wrong is in a position analogous to the position of the ISP customer who knew the fault was at the sender’s end: they have a correct hypothesis and no access to the system that would confirm or refute it.
The appeal mechanisms that exist alongside automated systems were designed for human decision-making and often fit automated decision-making poorly. The human decision has reasoning that can be examined and found wanting. The automated decision has parameters that produced an output, and the question of whether the parameters were appropriate for the specific case is a different question from the question of whether the output was correct. The appeal system can examine the output. It is often not equipped to examine whether the parameters should have produced a different output in this case, because the appeal system was not designed to interrogate algorithmic parameters.
Australia’s Robodebt is the case that made this visible at scale, but the underlying structure is not unique to that scheme or that jurisdiction. The United States’ automated benefit determination systems have been documented generating systematic errors in food assistance, Medicaid eligibility, and child welfare assessments. The Netherlands ran an automated tax fraud detection system that wrongly flagged tens of thousands of families, predominantly from ethnic minority backgrounds, for child benefit fraud—a system whose discriminatory outputs were the product of the proxies the algorithm was trained on, which reflected existing social patterns rather than actual fraud.
The algorithm did not intend discrimination. The incentive architecture that produced it rewarded cost reduction and processing efficiency. The discrimination was the output of those incentives operating through a method that the people who designed it did not adequately examine for the ways it would perform across the population it was applied to.
The epistemic problem here is not merely that automated systems make errors. All decision-making systems make errors. The problem is that automated systems at scale make errors whose pattern may not be discoverable through the system’s own data, because the system records decisions made and decisions successfully challenged, and the challenge rate is determined by the capacity of the affected population to mount challenges, not by the actual error rate. The population most likely to receive incorrect automated determinations is often the population least equipped to challenge them—least equipped because the same conditions that produced their need for welfare also reduce their access to the information, the legal support, and the administrative literacy that a successful challenge requires.
The system therefore has systematically incomplete feedback about its own accuracy. It knows what it decided. It knows what was challenged. It does not know what should have been challenged but was not. The gap between what was challenged and what should have been challenged is the gap in which the system’s epistemic self-assessment operates. The system assesses itself as performing within acceptable error parameters.
The acceptable error parameters are calculated from challenge rates. The challenge rates are determined by the capacity of the affected population to challenge. The system is measuring its accuracy against a denominator that the system’s own design has limited.
This is the most consequential form of the epistemic degradation these essays have been tracing. The system does not become cruel in any intentional sense. It becomes progressively less capable of knowing whether its outputs correspond to anything the system was established to produce. The outputs continue. The capability to know whether they are correct continues to narrow.
The need remains.
The system processes it.
The processing continues.
The correspondence between the processing and the need is the question the system has lost the mechanism to answer.
The Vorpel Foundation