Epistemic status: Obviously speculative sociology. Probably pretty obvious to some but I’m just trying to crystallize these ideas from my mind onto paper.
I was recently on a random walk through some old SSC posts and stumbed upon his review of Cyropaedia. His interlude about the ‘Fremen mirage’ and the blog by Bret Deveraux got me thinking again about the question of decadence and whether or not this is a real thing.
Most of the discourse around decadence seems to be people primarily attacking this meme and saying variously that decadence is typically a concept used to bemoan moral decline or attack present day things that they dislike, often from a rightist direction. People claim that decadence is fascist, right-wing, there is no such thing as decadence etc. This critique is often correct against the specific things that people post, but nevertheless I think there is a kernel of truth in the concept that decadence is pointing at. There does seem to be some kind of generalized pathology and ineffectiveness of groups (and maybe individuals?) that is caused by too much success and luxury, especially in competitive settings such as wars but also in e.g. intellectual output and production. We see many examples of previously successful groups and organizations slowly ‘rotting’1 over time and becoming vulnerable to those with vastly fewer resources than them. Small, cohesive groups appear to win against seemingly vast odds much more often than should be expected. This doesn’t always happen, of course, and probably not even the majority of the time, but it certainly happens more often than ‘always never’ which is what we should expect by default. We see this both in political history but also in business history where the cycles of incumbents being ‘disrupted’ by startups is well known. It also seems to appear in cultural history where some group of innovators create a new kind of style which then is copied and formalized, slowly becoming an empty mask of itself, before being replaced in its turn. We also see it in intellectual communities and religions where some theory or belief system takes hold, achieves the zenith of power/acceptance and then slowly declines in an almost imperceptible way, appearing strong on the outside but rotting within, until suddenly it is replaced by some new and vigorous upstart2.
Decadence has a bunch of connotations. Being weaker either physically or ‘morally’, at least on a per-capita level; being complacent and ossified and unable to adapt to and react to novel threats; being more concerned with image and appearance than reality3; taking cowardly short-term actions which doom themselves in the long term such as paying the danegeld or generalized dysfunction which results in similar outcomes; constant internal instability and backstabbing; a general malaise and lack of any kind of intrinsic loyalty or group cohesion with everybody out for himself even when this has globally catastrophic outcomes; the rituals of society keeping their form (and perhaps growing in grandeur) but becoming meaningless and empty. In some ways, this is a tale as old as time. “Pride goeth before the fall” etc.
Is any of this real? Should it be described by a single term such as ‘decadence’ or is it a bunch of unrelated issues which coincidentally occur? If it is real, what are its actual mechanisms vs vague moralistic speculation that gets wrapped into some historian or thinker’s idiosyncratic axe to grind? Assuming it is real, is it a historical phenomenon which ended, perhaps with the transition to the industrial era, or is it still there? Certainly, empires being overrun by barbarians is no longer a real threat due to modern industrial era weaponry, supply chains, and mass mobilization of militaries, as well as the fact that there are no obvious barbarians left.
The first question is can the whole concept of decadence just be explained by regression to the mean4? Due to luck, some organization or state becomes very large and powerful but then obviously their luck has to regress and so they perform poorly in future relative to their stunning initial successes, which presumably will eventually result in collapse if the ‘equilibrium success’ for whatever reason is substantially lower than their achieved success due to luck. This is certainly a possibility, however I don’t think this can be the full story. This regression can take many forms – regression of outside factors i.e. during the initial success period the world just happened to be uniquely suited to this kind of strategy, regression of the unique skills or traits of e.g. the founding generation which futures ones do not possess, regression more nebulously of ‘culture’.
However, the regression to the mean story completely ignores the strong feedback loops favouring incumbents that are in play. Regression to the mean is typically used to explain individual variations in performance rather than that or organizations, and only in cases where prior success does not unlock new skills or key resources for the individual. Trivially, we don’t expect an expert concert pianist who just did a perfect showing to revert back to the level of a random beginner because their entire success that night was ‘luck’ – they might be expected to revert to a slightly less perfect performance but still far above the random baseline. Instead, skill sets the mean and then we only expect fluctuations around the mean to be subject to regression. In the case of states and companies, success brings more resources that can be used to produce more success. This positive feedback loop exists for any entity that can accumulate resources to itself such as states and organizations and then productively deploy these resources to obtain further resources. An empire becomes stronger as it conquers more lands and obtains more resources and a larger population. A company becomes stronger as it gets more revenue and hires more employees. If an organization achieves a bunch of success, initially through luck, we should expect that in the next round of competition it is better placed and would succeed more without luck. That is, regression to the mean would not look like random walks around a mean of 0 but a random walk around an exponentially growing mean (if the returns to scale are positive and constant). Regression to the mean only really explains events when the underlying fundamentals are roughly fixed – for instance a competitor who does well at some sports competition who will likely do worse next season. The underlying latent variables (i.e. their talent and skill) are assumed to have stayed roughly the same but there is variance/luck and that is driving outcomes. This is not at all the case for historical empires or large successful companies. Historically, the world seems highly stochastic and slightly cyclic, with vaguely regular cadences of empires, companies, and other organizations rising and falling with their own characteristic half-lives.
The fundamental problem here is that at the ‘object level’ almost all the returns to success are positive. This means that naively success should generally beget more success in an exponential fashion. It requires an extremely powerful force to offset this default tendency. Regression to the mean is certainly not powerful enough for this. However, such a force must exist because, in practice, the world looks dramatically different from the simple exponential model. There is a constant random-seeming churn of states, empires, companies, wealth etc vs a world where the rich get richer indefinitely and thus everything ends up controlled by the winners at time t=0 forever after. We do not all live in the grand one-world Sumerian empire which simply grew faster and more successfully than its challengers during the hinge of history in 3000BC. Similarly, the East India Company no longer controls global trade; the Ford motor company does not have an eternal monopoly on cars; IBM does not reign supreme in computers. Similarly with wealth, despite its undeniable exponential properties, compound interest seems to be pretty weak in practice. The descendants of Croesus and Crassus do not lord over us all today5.
This means that there must not only be positive returns to scale and success but also negative returns. Larger states become harder to govern and more prone to civil strife. Larger companies become slow, unwieldy, and lose that indefinable ‘taste’ that made them stand out in the first place. There also appears to be some kind of phenomenon of increasing ‘entropy’ and decay in organizations or states that have been highly successful for a long time. Exponential systems are always poised on the edge of a knife destined to either explode or collapse, however the world largely seems to stabilize these systems or at least prevent explosions (there are a fair few collapses). Understanding and explaining these dynamics has to be one of the fundamental questions of sociology. Moreover, some of these dynamics seem to match our intuitive notion of ‘decadence’.
We need to try to understand what the mechanisms are behind this opposing force – the ‘curse of success’. Mathematically, to counter exponential growth we need a countervailing force of exponential costs, sufficient to flip the returns to growth from positive to negative, or at least less positive than potential competitors. These forces also need to be coupled to the size of the organizations somehow. Random shocks are not enough. Larger more powerful states or companies should be able to withstand exogenous shocks better than smaller ones. Instead, the scale of the costs needs to match the scale of the organization itself in order to match and eventually overwhelm the exponential growth.
Here, I discuss two potential mechanisms for this force to act:
- The first mechanism is that success tends to bring rigidity and diminished exploration due to higher global opportunity costs. In a nonstationary environment, where optimal strategies change often, this tends to cause diminishing competitiveness over time resulting in eventual diminishment and replacement by competitors.
- The second mechanism is inherently increasing communication, coordination, and internal misalignment costs which grow with scale and also over time in the form of increasing defection, parasitism, and ultimately cause a form of organizational cancer. In a dynamical system with long time-lags, this eventually causes overshooting, overextension, and ultimate collapse rather than simply equilibrium at the point where the marginal returns to scale are exactly zero.
To turn to the first mechanism, this is something that is most easily understood intuitively when applied to individuals and to companies. Every day individuals and organizations are faced with a wide variety of choices and tasks to complete. Many key tasks that are good in the long term are new, difficult, and risky. They are not necessarily super appealing at the best of times. If your day is filled with highly urgent and valuable opportunities in the short term, you can not pursue the long term properly. In fact it is irrational to do so. Only those with low opportunity costs can truly take immensely risky and low probability bets6. The larger your opportunity costs are, the greater are the costs to exploration or taking short-run costly, long-run maybe beneficial actions7. This even happens in RL environments. In most environments such as stationary bandits the optimal thing to do is have a decreasing schedule of exploration. At the start when you are unsuccessful and have no idea what you are doing, the optimal thing to do is to explore widely. But then later on when your policy is close to optimal, the counterfactual regret from exploring is high since instead of doing a random or mediocre thing counterfactually, you are comparing it to doing the optimal actions.
In Hamming’s terminology from “You and your research”, this is ‘working with the door closed’: more productive in the short run, but subtly deficient in the long run. Of course, the more valuable short term productivity is, the harder it is to make long term investments which require sacrifices to the short term productivity. If you don’t have much going on each day then you can do all the long term work you want and there are no costs to you of ‘leaving the door open’. However, if you have customers with billion dollar contracts each needing something done yesterday, and fifty direct reports who all need 1-1 meetings, then the opportunity costs of spending lots of time and resources on crazy long term bets, of which most will fail, is just massive and untenable.
Being successful effectively traps you in a web of obligations which are extremely appealing and lucrative in the short run, but which can render you vulnerable to long term shifts and make it hard to spot these changes. There is always a near infinite amount of short term lucrative things to do and extremely little incentive (in fact usually strong disincentive) to switch to longer term and highly risky things.
Moreover, focusing on the short run is rational. Most of those who venture outside fail and your odds are not really much better than baseline since you will be in an unfamiliar environment. You are already successful in your current environment and there is deep uncertainty with the new one. Who knows whether the new thing will work at all. Most likely it will not. Most potentially promising things end up flashes in the pan. Better to further improve what you already know works. Every step is prudent and rational until suddenly you are irrelevant.
Many companies, even when they know that the market is shifting away from them, prefer to simply maximize their power and revenue within their shrinking niche. Often this makes sense. Maximizing revenue within their niche is what they know well. Breaking outside the niche and doing something very different is extremely hard and risky. Very few people are incentivized by the long run and even those who are, maintaining the correct level of mental detachment to both prioritize short run wins while maintaining a grip on the long run is just extremely challenging to do. Additionally, from a pure economics perspective, simply maximizing returns within the shrinking niche is exactly what shareholders want them to do. Shareholders usually do not care so much about a particular company since they have invested widely across the economy. Instead of dying company A making a bold bet and potentially collapsing, much better for company A to just keep plodding along giving good but not amazing returns while they simultaneously invest in company B which is new and innovative and doing all sorts of crazy risky things. The shareholders can diversify across these risks while the management and leadership of company A cannot.
This idea is the core of Clayton Christensen’s disruption theory8. That incumbents have many advantages of scale, size, resources, etc, but their actual existing successes often become hindrances in the long run because they also entail a lock-in to the current way of doing things. The opportunity costs for doing anything else are just too high. And so, in a nonstationary environment, eventually there open up opportunities for newer firms, without high opportunity costs, to seize these opportunities, grow, and become the incumbents in their turn. These opportunities often (but not always) emerge at the lower end of the market, which the incumbents do not compete in as fiercely because the opportunity costs of further maximizing their existing business vs undercutting themselves to cater to the lower end are too high. This opens the space up to new entrants who can start there, establish a foothold, and then grow upwards.
Another similar story is told by Tanner Greer’s musings on public intellectuals. The question here is why public intellectuals only have a short shelf-life before their ideas fall out of circulation and they become objects of irrelevance at best, ridicule at worst. His hypothesis is that public intellectuals build a store of a few original insights while working hands-on in their field, in close contact with reality which almost always throws up observations that are strongly dissonant with the prevailing wisdom. If talented and lucky, our would-be public intellectual can synthesize their experiences and evidence with existing ideas to create a powerful and attractive new idea. They can then generalize this from the details of the specific place they found it to something generally appealing. By doing so they become a public intellectual, who comments on the general problems of the day, versus just an original academic specialist. They then achieve dramatic success and fame as a public intellectual and can milk this idea for a 5-10 year period where their insight is new and fresh and not too far removed from the world it originally described. Time marches on, however, and this idea is either shown to be false or made irrelevant by the flow of events, or else, if it succeeds, it becomes so commonplace it becomes a triviality. A new generation comes of age who are primed to think of everything their elders thought as declasse. And the influence and fame of the intellectual ages and fades.
Obviously most public-intellectual ideas speak to the zeitgeist of the times and so fade when the time moves by. However, the intellectual clearly has the skill of hearing the zeitgeist of the time so why can they not simply come up with new ideas when the old one fades? The argument here is that by this point, the public intellectual is far removed from the original deep and obscure work that originally gave insight. Instead they write general columns and appear on TV shows and podcasts and get feted by various important personages around the world. Thus, once their one idea fades, they are far removed from the generative process for the next idea and so are left with nothing.
But why can’t public intellectuals, who are generally smart people, recognize this and once they notice their well running dry, go back to the kind of thing they originally did and try again? Again the issue is opportunity costs. Once you have already achieved success, you have a huge range of possible and highly attractive opportunities always available. The opportunity cost of doing the things that originally brought some chance at originality and paid off in the long term has become too high. There are almost more podcasts and shows with vaguely famous people to do. Crucially, these are relatively easy and give a fairly certain pay-off. If you do the podcast, people will hear of your ideas and some will buy your books. Your name will continue to be in the news. You will be treated well and come out feeling smart and accomplished. If you go back to obscure hard research, you get none of this. Most of what you do is wasted and gives no or negative rewards. It is hard to do and feels bad mostly. Nobody will hear about what you are doing except, at best, a few who are similarly obscure. You will get no plaudits or applause and you will spend most of the time feeling like an idiot. Even after all of that there is only a small chance you will ever regain a sliver of the success you already have in your new domain. You are back in the territory of very low base-rates and lottery like success endemic to creative fields, and you are also much older, less hungry, and with less neuroplasticity than your new competition. At some point it becomes highly rational to just milk your existing successes and then retire vs trying again9.
Ultimately, the opportunity cost of sowing when you could be harvesting is too high, and so you end up never sowing until, of course, everything you have previously sown has been hashed and rehashed into oblivion and you are left with nothing more.
Even if you know this, it is still incredibly difficult and often irrational to go back to sowing, simply because of regression to the mean. There are lots of people toiling in obscurity ‘sowing’ ideas and insights that will never materialize or succeed. A-priori if you were previously a success you are probably better than average but luck likely also played a huge part in your success and lightning rarely strikes the same place twice. If you do go back to sowing, give up the speaking circuit and head back to the archive, your chances of coming up with a second great idea are at best only slightly better than a random PhD student doing the same thing. The rewards, while rich, are very far away and very uncertain. Much better to just glide into a gentle and comfortable retirement than go back out into the storm with only a faint chance of getting back into the warmth again10.
Coming back to the original notions of decadence meaning a lack of military fitness in Cyropedia, in Ibn-Khaldun, and in general parlance. This historical effect can also be explained by high opportunity costs. The classical story here is that the creature comforts of civilization make its men weak and effeminate. More interested in eating leisure, eating fine food, and wearing fancy clothes and engaging in social politics than in strenuous physical exercise, training military virtues, and going on conquering sprees. The reason the ‘barbarians’ did not do this is not because they chose not to but because they had no choice. They were forced into marginal lands only capable of supporting smaller sparser populations and which required intense physical exertion merely to survive.
Moreover, this was in an era before any kind of professional military training outside of perhaps a very small number of elite soldiers. There was also very little training in the intervening time between recruitment or conscription sending people directly into battle on campaign. This meant that the commander and state was stuck with whatever skills and state of physical prowess existed natively in the population at that time.
This naturally gave peoples from harsher marginal lands an innate advantage since they often had skills learnt (especially from hunting) that were useful in warfare. Sometimes they were also better physically conditioned than people who lived in cities (although not necessarily vs actual farmers from the richer areas). This was especially the case for all steppe nomads who learned their amazing cavalry and archery skills as a direct consequence of their steppe herding and hunting lifestyle, and which would be prohibitively expensive for settled societies to replicate by directly training people. In Cyropaedia, Cyrus tells a similar story about how the Persians gained their horse and archery skills from hunting. Similarly, even in the middle ages, England gained its unique population of highly skilled longbowmen from village poachers before various kings initiated explicit archery competitions in rural villages to deliberately train the population in archery.
However, importantly, living in these marginal lands and learning these harsh skills was rarely by choice. The reason they ended up in the marginal lands in the first place is usually because their ancestors had lost wars and had been kicked out of the rich farming districts. Given the opportunity, they would usually either leave and migrate somewhere better or else (and this often happened especially whenever the barbarians were ultimately successful). However they rarely had such choice and thus existed as large potential wells on the outskirts of civilized society, usually quiescent apart from occasional raiding activity, until somebody sufficiently charismatic and talented enough arose to create unity, forge an army, and go on a conquering spree. The barbarians therefore were effectively forced into the unpleasant, low-EV high-variance strategy while the ‘decadent’ settled civilizations chose (sensibly) the opposite.
The invention of professional militaries with actual full-time military training as well as the decoupling of military skills from civilian life have both rendered this form of decadence irrelevant in the industrial and modern eras11. The existence of large professional militaries and standing armies crucially depends on advancing agricultural and other technologies creating large enough surpluses that states can afford to keep around non-trivial proportions of the population doing nothing except training for war. Prior to the 19th century, this was not really possible except for tiny aristocratic elites. This means though that wars are no longer fought by untrained civilians relying on whatever random skills they learnt in their prior life but by well-trained and disciplined professionals which largely removes this advantage the ‘barbarians’ have over settled civilizations. Secondly, due to much greater specialization and capital intensity of warfare, the skills learnt outside a military role are almost entirely irrelevant. Militaries today rely entirely on training12. A random person conscripted on the street is highly unlikely to be able to zero-shot operate a SAM battery, drive a tank, dogfight in a jet, perform long range sniping, or pilot FPV drones. To train somebody in these skills requires a level of surplus and capital intensity that is only achievable in the most advanced societies. This version of decadence is therefore no longer a going concern, however the general mechanism of increasing opportunity costs leading to less exploration since it becomes more costly leading to decline and increasing irrelevance is a much more general mechanism that is still in operation today.
The second, and perhaps more fundamental mechanism, is that as organisations of any kind grow, the internal coordination and communication costs rise. Simultaneously, misalignment between the individual and the organization increases because internal competition eclipses external competition in importance to the individual. This means that internal defection (vs cooperation) becomes more favoured at the same time that monitoring mechanisms become much more challenging due to scale. Ultimately this results in larger organizations seeing reduced or even negative returns to scale as the coordination and communication overhead needed to grow outruns the positive returns to scaling. Over longer time horizons, internal defection and misalignment also grows within the organization causing effectiveness to decline further, albeit with a lag, possibly resulting in a sudden collapse.
The first point is fairly obvious. Organizations are made up of people. Even with the best will in the world, for an organization to act coherently, it must be able to communicate and coordinate with all its constituent members. As the number of members grows the amount of communication must increase. Super naively it increases quadratically, which is infeasible. The theoretical optimum is logarithmic which is why most organizations end up with a tree shaped hierarchy. However, the tree has its own limitations of information propagation down the levels being very far from lossless, as information ends up being distorted heavily at each layer, meaning that the optimal communication scheme also involves global broadcasts which are at least linear. Given that returns to scale are usually linear if not slightly sublinear at an objective level then this means that to continue to scale the communication costs can at best be linear. In practice, coordination and communication also takes time which also serves to increase the ‘latency’ of an organization’s responses which can also impose substantial additional costs especially in adversarial scenarios.
However, communication costs alone cannot be the full story13. Optimal communication patterns such as trees or hierarchical small world networks are logarithmic or at least sublinear. Additional mechanisms are needed to reduce returns to scale to zero or turn them negative. Beyond simple overhead, large organizations additionally seem to suffer from some kind of slow ossification and outright decay. This arises from a more pernicious mechanism. As organizations grow, for individuals within the organization success comes primarily from winning internal vs external competitions. This means that internal misalignment and defection is increasingly selected for at the same time that the usual oversight mechanisms to prevent this are becoming more costly due to greater communication overhead.
For organizations made up of independent ‘selfish’ agents, this natural shift from external competition to internal competition makes sense and is entirely rational. When an organization is small, group selection dominates. You will mostly end up successful if your organization succeeds, otherwise you will end up in a bad situation. There is no point being a senior manager of a dead company, or an aristocrat in a destroyed state. However, once the organization grows beyond quite a small value, some level of continuity and success is assured. At this point, status maximization within the organization becomes the rational thing to do. This is true even if an individual agent is actually somewhat selfless and cares about the organization as a whole. In this case, to actually pursue good policies, the agent still primarily needs to win political battles against internal foes rather than actually engage with external reality. For any goal, even aligned ones, the importance of internal-facing competition vs external-facing increases with scale14.
Although this often results in bad overall outcomes, from the personal perspective of anybody within the system it is highly rational. Imagine you are a Roman Emperor. In this case, your primary enemies personally are not actually the global enemies of the empire. Instead they are primarily other Romans who can threaten your power. You are rationally correct to fear and try to destroy any successful generals who might attempt a coup, even if it means severely degrading your own military. After all, if some remote provinces gets overrun by the Goths, does it matter, really? Certainly, none of the Roman aristocrats at your Palatine Hill dinner parties seem to mind very much and the impact on your day to day life is minimal. However having a charismatic, successful, and popular general is a direct personal threat to you since he could easily raise his legions in revolt, march to Rome, and depose and murder you and your family, as happened to the last five emperors before you15.
Although an extreme example, this principle applies everywhere. If you are a middle manager in some generic large corporation, the primary determinants of your advancement and success are how well you are thought of by the middle managers above and around you16. Their opinion is (hopefully!) somewhat correlated by how effective you are at achieving the general goals of the organization but is much more affected by specific interpersonal friendships and alliances you can make, projects that you can claim credit for, the features you(r team) have launched, and your supposed ‘impact’, and generally how well you are at managing bureaucratic processes and navigating complex social systems.
This process proceeds fractally to everybody in an organization. As organizations grow bigger, the fitness and ‘success’ of an individual within the organization becomes increasingly uncorrelated with the success of the organization as a whole. Free-riding becomes substantially easier, since the monitoring becomes harder at scale and the costs become less noticeable. Communication within the organization becomes more challenging and distorted. Individuals or groups of individuals realize that it is better for them to pursue their own interests rather than those of the organization as a whole and can create psuedo-nepotistic fiefdoms inside the organization which grow while subtly undermining its effectiveness and siphoning off its resources. The primary locus of conflict becomes internal and social, between people jockeying for position and power within the organization instead of by solving the problems brewing on the outside.
We can model this as a kind of iterated evolutionary prisoners dilemma. There is some population of defectors and cooperators. At the start of an organization, monitoring is high and so defectors can be detected and removed. Also success primarily comes from the group as a whole succeeding so that cooperation is twice advantaged. As the organization grows, the group selection pressure declines and the individual competitive pressure increases, leading to internal defection being increasingly selected for. For some range of sizes this pressure towards defection can be kept in check by monitoring and reputation systems. However, as scale continues to increase these defences weaken and defection increasingly becomes selected for and grows unchecked. Eventually the growth of ‘defectors’ such as parasitism, internal misalignment etc grows fast enough to consume the surplus generated by positive returns to scale and so the returns to scale tip negative. At the organizational level, this causes decaying effectiveness which, in the best case, stabilizes at some low level where the internal processes and defences within the organization against this pathology, as well as the need to actually interface with the external world to keep things alive equilibrates with the growth of the internal cancer. In the worst (and perhaps usual) case, there is a substantial time-lag between the initial creation and growth of the surplus and the growth of defectors to consume it. This means that the organization grows substantially larger during the initial mostly defector-free period than the ‘true’ ‘fully-parasitized’ equilibrium returns to scale would admit, causing a period of increasingly negative returns to scale and thus ultimately a diminishment or collapse of the organization as a whole.
We can think of this almost as a Lotka-Volterra predator and prey-style model. Initially there are a bunch of resources and free energy so the cooperators (‘prey’) expand rapidly to eat it up harvesting the positive returns to scale until they get sigmoided by increasing communication and coordination costs, however during this period large surpluses are created. Then during this slow growth period, the organization has become large and defectors (‘predators’) begin to quietly begin to get selected for and grow. As the internal mechanisms to prevent defection become weaker both due to scale and also due to increasing numbers of defectors (some of whom might be tasked with executing these mechanisms!), the number of defectors rapidly rises. In the continuous time instantaneous case, this forms an equilibrium where the cooperators are producing just enough surplus for the defectors to consume where the cooperators cannot grow because of the defectors and simultaneously the defectors cannot grow because if they do there no longer become enough cooperators to support their higher population. In the time-lagged case, however, this instead produces dramatic boom and bust cycles which more closely match what we tend to see in reality.
Of course this is just a theoretical model and the actual reality it purports to represent is much more messy. However, I think these mechanisms provide very general models and explanations of the kind of diminishing effectiveness and progressive ‘rot’ that seems to afflict large organizations over time. I don’t think this is an exhaustive enumeration of all the forces that in practice overcome the intrinsic positive returns to scale but these certainly seem to be two plausible candidates for them. Generally, it is interesting to think about whether there are any other mechanisms that could also form part of this ‘curse of success’ and also how the strength of these forces is likely to evolve over time and be affected by radical technological changes such as AI.
-
See Kuhnian paradigm shifts, and Collins’s sociological model of philosophy. ↩
-
Cf the Emperor’s new clothes as a classic sign of decadence. Any Emperor who had obtained his throne through personal conquest would never be so absolutely out of touch. ↩
-
This is the Hansonian view of cultures regressing to the mean. ↩
-
In the case of wealth compounding I’ve had some thoughts on this previously which are relevant. ↩
-
We glamorize the people who are forced into low probability super high variance bets, because some time fraction of them win. In fact, however, you are almost always better off not doing this as the actual EV is low there are just strong selection effects whereby we only here the successes. It’s a similar story as to why ancient people glamorized the ‘hard’ barbarians who lived in the wilds without any of the comforts of city living. The cultivate their cultural or military virtues precisely because they are forced to. Given the choice, most of the barbarians would want to urbanize and live in cities, as eventually happened. ↩
-
More speculatively, it seems plausible that something like this is the cause of plummeting fertility worldwide and especially in the developed world, seemingly not despite but because of greater wealth and opportunities. In effect, due to greater wealth and economic opportunities for women, as well as vastly superior entertainment options, the opportunity cost of having children in terms of income and experiences foregone is much greater now than in the past. Since women now have the technology to control their own fertility, this results in increasing numbers delaying childbirth until later or never at all resulting in pretty much the entire developed world and now much of the developing world being sub-replacement fertility and potentially dropping to even lower than that. Modern technology and economics makes the opportunity cost of children simply too large for many people to bear. ↩
-
Interestingly, the fall of the Roman Empire can be cast, at least in part, as a case of disruption theory. The roman empire did not expand to conquer the lands held by the barbarians such as the Germans and the Goths because it was too poor – i.e. ‘low end’ – that the opportunity costs of actually owning and administering it were too high for the state to annex it (at least in the short run. Probably over many generations it would have become richer and net-positive, like Roman Britain). Because of this, space was created for new entrants like the Germans and the Goths to establish their own proto-states and learn Roman technology and tactics which eventually, in moments of weaknesses of the incumbent, they used to ‘move upmarket’ and eventually displace the incumbent entirely. ↩
-
This is a situation where regression to the mean is highly applicable. ↩
-
This is something I also notice in my own life at the micro-scale. Certain activities such as reading papers, running experiments, fighting bugs, and trying to attack specific problems, and even often going on random walks through blog-space end up eventually generating new ideas but with a long lag time, high levels of stochasticity and frustration, and of course large degrees of uncertainty. Some idea you have might be sparked by reading two papers a few months apart and then you might be able to get it working quickly because you built the skills to do so by implementing some other experiment which failed miserably previously. Of course, until this insight occurs (and there is absolutely no guarantee that it ever will!), all you see is a bunch of frustrating failures and dead-ends that appear to go nowhere. The path is hard and only ever visible in retrospect. Doing these activities effectively builds and stores ‘serendipitous insight’ which may eventually come out at some point in the future. The vast majority of time spent doing these things is wasted though but there is no way to know ahead of time what will be valuable and what will not be. Conversely other tasks such as writing blog posts, doing final experiments, writing papers etc use up this store of serendipity. Finally, there are classes of tasks such as doing meetings, handling admin work, handling routine experiments etc which is neutral – which takes time and often gives immediate rewards such that you feel you have had a productive day after ticking off a bunch of them but have little long term rewards. It is too easy, especially as you become more senior and demands on your time increases to spend almost all your time on this latter set of tasks – which often feels like ‘becoming more productive’ because there are no more days spent playing with things that are obviously doomed in retrospect or just reading totally random and irrelevant papers – but which causes the well of insight to eventually run dry. The opportunity costs are real though and this is something I already struggle with. Meetings and admin work is not created just to keep people busy. It is vital to the proper functioning of organizations. Somebody needs to do it and that somebody must have all the correct idiosyncratic context necessary to keep things running. Almost always, the only person with this context is you. And so the cycle begins anew. ↩
-
One place where this is not so irrelevant is the intrinsic opportunity cost differential of attackers vs defenders. For instance, if you are a mongol warrior under Temujin in his invasion of China, no matter how far you have to go and how much you have to fight you have pretty much no alternative but to stick with him. If you desert you would be alone, in a hostile country, unable to speak the native language and physically look obviously different from the Chinese, with both the chinese and the mongols trying to kill you. Better only stick with one enemy. You presumably also signed up (semi-?) voluntarily but even if you somehow made it back to Mongolia (several thousand miles away!) all you would return to is the life of crushing poverty as a herder on the steppe. By contrast, if you are a Chinese peasant drafted to fight Temujin, you are fighting in your own country, surrounded by people who speak and look exactly like you. If you desert then there are a load of friends and friendly people to help you. You have a family and a farm back home who are missing you and need your help for the harvest, and so on. The incentives are dramatically skewed towards Assabiyyah for the invading force and for defection and disunity for the defender. This especially becomes the case right when the fight becomes hardest and most hopeless-seeming and morale is more important than ever. ↩
-
Of course today anybody can just grab an AK47 and form an untrained military but in practice whenever such militia forces go up against the armies of strong advanced states the result is almost always extremely lopsided. ↩
-
Although the lack of fidelity of human-to-human communication especially across many hops is also a clear cost which increases communication burden and reduces its effectiveness. ↩
-
In some sense this is just part of a general principle that volume scales at one higher power than surface area. As scale increases, the fraction of internal interactions also rises. ↩
-
This is one of the reasons that dictatorships and authoritarian states and especially those with a history of coups often have surprisingly poor militaries in practice. The dictator is vulnerable and must select higher-ups in the military almost wholly for loyalty and not competence where often competence is anti-correlated with loyalty; if you are an idiot elevated far above your station you know you must be loyal to your patron for you have nothing else to fall back on. However if you rose up on your own merits alone, might these merits not let you ascend to even higher echelons? ↩
-
Zvi’s moral mazes sequence goes into great detail on this point. ↩