Is Endogenous Exploration Sufficient?

Epistemic Status: A short note which posits a question without necessarily arriving at a definitive answer.

When thinking about whether we are likely to end up in a monotheistic or polytheistic AI future, we often end up trying to identify and categorize the advantages and disadvantages of centralization vs decentralization. This balance, in the limit of alignment technology, seems likely to determine the scale of coherent entities in the long-term future. The advantages of centralization are usually obvious – better coordination, internal cooperation, the ability to marshal more resources to coherent directed ends etc. The advantages of decentralization are generally less obvious. Typically these include things such as challenges coordinating and internal communication bottlenecks, principal agent problems, internal defection and cancer, local information, distributed error correction and avoiding mass correlated failure modes, notions of general and adversarial robustness and so forth. Broadly, I think these separate into two rough clusters. Firstly, there are your classic coordination problems. We discuss these in depth in the decadence post, but in general, if a system is composed of many subsystems, it needs a method of binding the individual agencies of each subsystem towards the coherent long term goals of the higher level system. When this fails we get all manner of interesting failure modes which appear to exert hard constraints on the maximum size of coherent systems in our world and our history. As we recently posited, greater communication and coordination technologies, and ultimately successful AI alignment should largely remove these sources of failure, thus enabling the diminishment of lower level agency with the parallel construction of higher level ‘super-minds’, ushering in a transition analogous to the original multicellularity transition.

However, classic ‘alignment failures’ are not the only class of disadvantages to centralization¹. There is another class of costs of centralization which I will broadly name ‘informational and exploration costs’. Here the idea is having many agents is not just the equilibrium because coordination/alignment imperfections limit the size of the super-entities that they can meaningfully form, but rather there are also strong countervailing benefits of decentralization. These benefits come from a variety of putative sources. In a bunch of systems mere variance is vital. For instance, variation is the fundamental source of evolutionary change and generally biological heterogeneity provides a kind of innate adversarial robustness to biological populations. For instance, monocultures are highly vulnerable to parasites and predators while more heterogeneous populations do better because the parasite or predator has to evolve to target a wide distribution of phenotypes vs innumerable copies of exactly the same phenotype with exactly the same vulnerabilities and weaknesses. In evolution more broadly, obviously heterogeneity introduces variance into the population which selection can act upon which is beneficial in the long run. Similarly, in economics we get notions of Hayekian local knowledge and Schumpeterian entrepreneurship and creative destruction. For Hayek, a bunch of small firms will perform better overall than large central firms because the small local firms each have a bunch of specialized practical metis specific to their location or customer base where a larger firm cannot directly access this information or, even if it could, it would need to average it out to produce products suited to the average case but not niches. Similarly, economies need entrepreneurs to start a wide range of products, for which the vast majority will fail, but some will succeed and end up moving the economy forward. By contrast, large firms cannot similarly produce a bunch of highly experimental products which will mostly all fail. Again we see the same pattern re-occur in machine learning. Firstly there is the classic exploration vs exploitation problem. An agent learning what to do in some environment must choose whether to ‘explore’ and try something new, which is likely to be bad, but could potentially be good, or else simply do what it has learnt worked well in the past. In the short run, exploration is usually bad since once an agent is at least somewhat decent at its environment, then most unknown ‘directions’ are worse than the current equilibrium. However, in the long run, the agent needs to explore otherwise it will get stuck forever in the first local optimum it encounters. Secondly, sometimes in large-scale RL, having populations of agents being optimized, each with different strategies, proves beneficial over a single agent. Although now older work, the AlphaStar² results which produced highly capable game-playing agents, which still have not been matched by LLMs (!), all used large-scale population training over hundreds or thousands of separate agents/policies competing in a virtual league. This provided the necessary diversity to maintain exploration at the policy level vs individual agents collapsing to some degenerate local Nash equilibrium. Although they have also somewhat fallen out of favour, ensembling methods provide a similar benefit. Here, in cases with very little data, training multiple models on the same data and then averaging their performance gives consistent and solid gains to generalization. This works even in LLMs.

For concreteness, let’s separate these out into two classes:

– Exploration Benefits: Many agents provide an inherent diversity or variance into the world which is broadly beneficial in the long run. For instance, agents can exhibit different phenotypes – some being highly ‘exploratory’, which are worse on average but higher variance, while others are ‘exploitatory’, who do better on average but with much lower variance³.

– Information Benefits: Many local agents maintain some irreducible stock of knowledge based on their local situations and phenomenology which results in better/more effective systems overall.

If we posit that coordination costs can be solved through better technology both through communication technology and also AI alignment techniques, then the only remaining benefits to decentralization remain these exploration and informational benefits. The question, then, is whether these benefits can only be obtained through a decentralized scheme or whether, in the limit of technology, a highly centralized singleton can effectively architect and operate itself such that it can gain these benefits while also remaining fully centralized – i.e. whether endogenous (to the singleton) exploration is sufficient to emulate the kinds of benefits that you naturally get as part of a more decentralized system?

This is the fundamental question, and I think the answer to this will be very important to determining the effective power and ubiquity of singletons in the long run. As we note in the autarkic agency post, even if we start out with a highly polytheistic society of many competing/cooperating agents, as technology improves, especially communication technology, then the economic logic of trade leads towards outsourcing increasingly large amounts of cognition and autarkic agency and becoming increasingly subsumed in larger systems, which themselves become increasingly networked together. Over time, these may behave indistinguishably from the internals of highly coherent singletons, so we effectively get a slow and subtle collapse to a singleton rather than a fast collapse, but we end up pointing towards the singleton nonetheless⁴. However, if there are direct countervailing benefits to decentralization, then we should expect this independent agency to be preserved in the equilibrium with the ratio depending on the relative strength of these benefits vs those of centralization.

So, to what extent do we think that endogenous exploration is sufficient? Obviously this is a massive question which we cannot hope to answer here, however we present some preliminary speculation.

Broadly, in the long run I am pretty skeptical of endogenous exploration not working. The limit of technology just seems too powerful. For instance, increasing communication bandwidth should slowly reduce the scope for irreducible local Hayekian metis. This is basically just a rehashing of the original central planning arguments. Obviously, in the early 20th century, central planning at the level of the economy was intractably difficult with their technology. With today’s massive computer infrastructure, this kind of planning can be vastly more tractable, efficient, and responsive, although it is unclear it can scale as well as capitalism to the whole economy. However, many internet firms effectively do central planning over huge swathes of economic activity and this is clearly workable to some extent. With far-future communication capabilities plus the computational capacities of extremely large ‘super-minds’, extremely detailed local information should be easily catalogued, correlated, and integrated. There will still be a problem with latency for coordinating over long distances, but this merely requires some kind of hierarchical small-world topology of the global mind with various timescales of communication, and although it prevents global coherence of information, it does not prevent a massive degree of local information pooling, and the same synchronized high-level value function being applied globally.

The second question concerns the exploration benefits – can a singleton explore sufficiently to match the intrinsic exploration of a decentralized system? In our current world, we see this question crop up time and time again in different guises, and usually the answer there is no. A big company can only extremely rarely incubate startups as effectively as the outside economy. Individuals struggle to explore intellectually or in their personal life as much as an ensemble across all other people can. Individuals in e.g. science and other fields tend to end up pigeon-holing themselves after success rather than continuing to deepen exploration forever. We struggle to make monocultures adaptive enough to handle the levels of adversariality that nature sometimes throws at us which regular heterogeneity can solve just fine.

However, most of these issues seem to be primarily ones of either technological competence and bandwidth or else standard alignment issues. The monoculture argument is already sometimes and often mostly solved by today’s biotechnology capabilities. For instance, we can treat many diseases which monocultures are vulnerable to by large amounts of pesticides, herbicides, and other chemical and biological agents without needing to introduce a huge amount of phenotypical heterogeneity. What this means is that heterogeneity and exploration at the level of phenotypes is not the only defense. Often having alternative mechanisms that depend upon the scaffolding of the wider civilization prove much more effective.

For why firms cannot incubate startups effectively, a lot of this is a combination of difficulties on the firm-side in that actually creating truly independent startups which can go off and act in opposition to or cannibalize parts of the firm’s core business is very challenging at an organizational level, even when this is best in the long run. Secondly, the incentive structure is typically wrong on the incubated-startup side too. It is very challenging/defeats the point of the incubating firm to give massive equity amounts to the startup’s founders/managers, however, this equity is the primary motivator for the startup founders/managers to begin with. The interests of the firm and the incubator founders are thus in direct conflict and this is usually irreconcilable. Alignment technology would prevent this by overriding the individually selfish conflict of interest that occurs here. Similarly, it would override the short-term pushback about cannibalization etc with a recognition of the longer term benefits. A second issue is that an incubated startup must maintain some degree of independence from its evaluators at the incumbent over short-to-medium time-horizons. As the aphorism goes, most good startup ideas seem bad. There must be some notion of slack and independence for people to pursue ideas that seem bad for a long time, while somehow avoiding the adverse selection of just incubating endless numbers of bad ideas. This is a hard problem to balance, even with the best of intentions, but nevertheless, organizational structures for this can be designed specifically and in practice startup incubators do often appear to work decently well.

For the individual person failing to explore, a lot of this is to do with either being strongly limited by opportunity costs – people do not have infinite time and hence cannot meaningfully explore anywhere near the variety that the full ensemble can experience – as well as the strong and unusual degree of fixedness of differences in personality and aptitudes between individuals. Presumably, our hypothetical singleton would not have these issues. Any sensible singleton AI system will likely be universally plastic, like current AI systems seem to be. Secondly, the opportunity costs for the singleton will certainly exist, but will be lesser, since the singleton will have vast computational resources to explore many paths in parallel that individuals today do not possess.

Perhaps the deeper question concerns the structure of the noise that underlies this intrinsic exploration of decentralized groups. Exploration via evolution or directed exploration of individuals that can then be ensembled can be conceptualized as some set of perturbations from the current policy. In the case of evolution, this is fairly obvious. We can think of some kind of mean population phenotype, while individuals are mostly deviating from the population mean through a fairly well-known and isotropic noise process. I.e., they have some set of mutations in various places in the genome, where the mutations are mostly independent. Evolutionary theories of e.g. drift are usually successful with fairly basic and trivially modellable noise processes. This means that if a singleton wanted to simulate the noise process of evolution internally, it obviously could. To me, this implies that a singleton should certainly be able to endogenously generate evolutionary-style variance and exploration if desired.

Exploration via directed intelligence of individuals could conceivably be much harder for a singleton to model internally. This is the case of things like entrepreneurs in some diverse society going off and spending large amounts of directed cognitive effort trying to invent new ideas, most of which are doomed, but some are successful enough for this whole endeavour to be beneficial on-net. This kind of exploration can penetrate substantially further away ‘in latent space’ from the current ‘state’ of the singleton. Moreover, because it is intelligently directed, its exploration profile is substantially more efficient than the kind of random walk that naive evolutionary-style mutational noise produces. This means that much deeper paths can be cut through the state space for a fixed amount of effort, time, or computational resources. This kind of exploration, then, is enormously useful and productive. The question then is to what extent can the singleton somehow either perform this kind of directed exploration ‘itself’ or can instantiate subminds which can do this exploration but in a manner that does not just result in the dissolution of the singleton itself – that is, a singleton must maintain sufficient internal diversity that permits substantial serial divergent exploration, but it must periodically re-merge and consolidate its internal paths such that it does not splinter and decohere itself. The question is whether this kind of carefully stage-managed phases of internal exploration and recoherence are actually possible in the long run without either collapsing in novelty and exploration or the singleton itself decohering over time.

Another way to think about this is that the core issue to solve is the broader problem of correlated failures within the singleton. Even though the singleton could be exploring, its own exploration mechanism could be subject to this kind of correlated failure mechanism – i.e. it explores but all explorations end up following the same ‘known-good-for-exploration’ paths, which thereby miss the actual important exploration in the long run. To some extent, then, the question becomes essentially whether a singleton can have sufficient foresight, wisdom, and general internal intellectual diversity to cover enough of the possibility space to insulate itself from unknown-unknowns. The framing of unknown-unknowns implies no by definition, since the whole point is that the unknown is unknown, however the counterargument here would be that in the limit of sufficient technology and intelligence, then the entire possibility space is actually known and mapped at least well enough to rule out any true unknown unknown from occurring. This is also a dynamics question. Typically these kinds of correlated failures come about through a recursive cognitive loop that bends back in upon itself – perceptions are filtered through a shared ontology; what is chosen to be measured and perceived depends on the model of the world; actions can be deployed based on plans that are based on the world model, and can themselves shift perceptions. The structure of cognition is necessarily a recursive loop and this loop can easily become pathologically self-circling and self-confirming. A sensible singleton, therefore, retains a strong residue of self-adversariality. Its thoughts and planning must branch and cover volume broadly rather than collapse into a single stream, and moreover, its pattern of thought must, at times, be self-repelling. Simply having thought down a path before implies that new thoughts must avoid that path and take another. Self-repellingness is not enough, however, to be efficient exploration must also be directed to avoid exploring clearly bad regions of the state space, where these bad regions actually comprise most of its volume. More importantly, this self-repellingness must also take place at the meta level of evaluation. Presumably there would be some machinery to cut out paths that the evaluator thinks are bad, to avoid wasting resources, however, the evaluator itself can have correlated failures. This then cascades down to correlated failure to explore at the object level. This means that the singleton must also provide an ensemble of evaluators, representing the true uncertainty over the evaluation metrics in which to judge paths, and ideally perform a similar exploration mechanism of self-repelling directed exploration in value-space as well as in the object-level state-space⁵.

One interesting advantage of singletons here is that this self-repelling property can be globally coherent and efficient in a way that naive decentralized exploration is often not. In real decentralized systems and markets many explorers can all have the same idea and effectively duplicate exploration efforts. Moreover, they often ignore, forget, or just be ignorant of prior feedback that a path is doomed, which is obvious ahead of time from the global view. A singleton could theoretically avoid these pitfalls and hence practice more efficient exploration. The danger here though is that the singleton itself becomes trapped by its own prior judgements and again falls to unknown unknowns at the meta level – i.e. it slightly incorrectly judges some paths to be unfruitful but which through some missed factor suddenly become valuable again. The question then is whether the cost of these meta-level failures outweighs the inefficiency of naive decentralized exploration, and secondly whether the singleton has meta-level exploration algorithms that can patch this. Certainly these exist, at a naive level introducing some irreducible noise provides a very inefficient method of always eventually exploring such unknown-unknown paths by sheer accident. This produces a kind of inverted bias-variance tradeoff which is mixed with efficiency. Raw variance is the ultimate backstop of total exploration but can be exceedingly inefficient; directed exploration is vastly more efficient but produces various kinds of biases.

The core question that thus faces the singleton is effectively the construction of an optimal self-exploration algorithm that can track and resolve uncertainty at all the meta levels. In one sense, this is already solved: Bayes-optimal exploration is a known program⁶. The trouble, as always with these things, is in the actual implementation details – how to actually track uncertainty not only in the world but in your model, and then how to solve the recursive problem of tracking the uncertainty in your uncertainty estimator, and so on. Secondly, the true Bayesian optimal solution is of course computationally intractable, and so the game becomes finding decent enough approximations that scale well with compute. The question is then whether a singleton can find such exploration algorithms that outperform the more ‘natural’ kinds of exploration intrinsic to decentralized systems. Again, I lean towards the answer being ‘yes’ here – civilization in some sense is the long road of producing mechanically designed systems which vastly outperform their natural equivalents – however clearly this is uncertain so I don’t have a definitive conclusion.