Indirect lessons from human alignment
On doing what evolution wants us to do
(Inspired by a post from Eli Tyre.)
Many people make some variant of the following argument:
Evolution is an “outer optimizer”. It is trying to make us maximize reproductive fitness.
We are “inner optimizers”. We just do what feels good.
But what feels good has been set by evolution, which is hoping that it will make us maximize reproductive fitness.
But we don’t maximize reproductive fitness.
In fact, birth rates are dropping everywhere.
Therefore evolution failed.
The standard interpretation is that this shows that alignment is hard. We have one example of an attempt (by evolution) to align the behavior of an intelligent system (us) towards some goal (maximize reproductive fitness). And as soon as that intelligent system (still us) was put in a different environment (modernity) it failed to continue to pursue that goal (you reading existential angst+science blogs instead of making/nurturing babies).
To be clear, it’s likely good that evolution failed. A world where everyone woke up every day and threw everything they’ve got into maximizing their number of descendants sounds grim. But say that you want to build a new intelligent system and tune it to do what you want. Will it keep doing what you want after circumstances change? The one example we have says: Maybe not.
But perhaps we can learn more from this example. Say your friend Alice does something. Maybe she buys a grapefruit or starts hosting a weekly board game night. If you ask her why she did that, she’s unlikely to say, “I thought it would increase the number of my genes that are recursively present in future generations.” Instead, she’ll probably say that it advanced some simpler goal like “not being hungry” or “fun”.
That is to say, evolution didn’t just try to align us to maximize reproductive fitness: It created sub-goals and then tried to align us to those sub-goals. Maybe this can give us additional clues about how hard alignment is? Maybe we can break down the question of, “How successful was evolution at aligning us to maximize reproductive fitness?” into:
How successful was evolution at decomposing reproductive fitness into simpler sub-goals?
How successful was evolution at aligning us to these sub-goals?
Problem 1: Does this even make sense?
Here’s a problem: It’s not obvious that this way of thinking isn’t pure gibberish.
When we say that evolution “tried” to optimize reproductive fitness, we are speaking in a kind of code. What we really mean is: You either create more copies of your genes in the next generation or you don’t. If you do, then the number of copies of those genes in the gene pool goes up, and they get more chances to copy themselves in following generations. If you don’t, then they don’t. This is almost literally an optimization algorithm running in an outer-loop, with our lives in the inner loop.
(Whenever someone talks about evolution “trying” to do something, there is lots of moaning about their naive teleological thinking. Evolution can’t “try” to do things, because evolution is not an agent and does not have goals. That’s true, but I find it somewhat pedantic, because there’s no other equally concise way to talk about this optimization. Let’s just stipulate that we’re using the word “try” in a specific technical way.)
Fine. But what do we mean when we say that evolution “tried” to optimize some sub-goal? You probably feel good when attractive people laugh at your jokes. But say you’re great at getting attractive people to laugh at your jokes but never reproduce. Whatever genes helped you do that will not spread.
By my lights, this objection is simply correct. There is no optimization for sub-goals. Evolution cares about reproductive fitness and reproductive fitness only. (Though see Kaj Sotala for a somewhat contrary view.)
At first, I thought this doomed this whole project. But suppose that while aligning us for reproductive fitness, evolution just so happened to align us to stay away from rotting smells. Isn’t that strong evidence that if evolution had tried to align us to stay away from rotting smells, it would have done at least as well?
If evolution failed to align us to some sub-goal, we can’t say much. Maybe it failed because alignment is hard, or maybe it “failed” because that sub-goal wasn’t important. But if it did manage to align us to some sub-goal, then it’s OK to treat that as evidence of alignment success.
Problem 2: What sub-goals?
Suppose I made the following argument:
Modern people are well-aligned to spend lot of time watching short-form video on their phones. Therefore it’s not that hard to align people to spend lots of time watching short-form video on their phones.
Something seems wrong, no? Surely all the time you spend watching short-form video represents a failure of alignment? On the other hand, suppose I made this argument:
Modern people are well-aligned to avoid starving. Therefore it’s not that hard to align people to avoid starving.
Technically speaking, evolution doesn’t care if we starve. If starving to death helped us have more babies, we would presumably be delighted when we starve to death. But in reality, it doesn’t. So, intuitively, this argument seems OK.
The problem with the first argument is that it paints the target around the arrow. The second argument is more convincing because it’s based on a durable subgoal that was strongly related to reproductive success in our ancestral environment. If we want to learn about how hard alignment is, we should restrict ourselves to subgoals like that.
So what subgoals do people have? This turns out to be a whole sub-field in psychology. It seemingly began in 1943 with Maslow’s famous hierarchy of needs. After poking around this literature for a while, I decided to adopt the model of Kendrick et al. from 2010, which is explicitly based on the relationship of goals to reproduction. They list the following:
Immediate physiological needs (Air, food, water, cold, heat)
Self-protection (Avoid violence and accidents)
Affiliation (Have friends and family)
Status / esteem (Be liked and respected)
Mate acquisition (Spend time and have sex with charming attractive people)
Mate retention (Keep those charming attractive people around)
Parenting (Nurture cute things)
These seem reasonable.
So how did evolution do?
Let’s suppose that evolution tried to align us to those sub-goals. That is, let’s suppose that in our ancestral environment, people who were good at pursuing those sub-goals tended to reproduce more, meaning that there was evolutionary pressure in favor of genes that make us care about those sub-goals. How will did that alignment generalize to the present day?
To answer that, I made up some numbers. That is, I subjectively scored each of those subgoals on a scale of 0 to 10, where 0 means modern people completely disregard it, and 10 means we pursue it strongly as we did in our evolutionary past.
Immediate physiological needs: 9.5/10. We remain extremely interested in not freezing or starving to death. The only reason I don’t give this 10/10 is that most of us don’t eat that well, meaning our alignment to eat in a way that promotes health doesn’t translate perfectly to the modern food environment.
Self-protection: 9/10. We remain very interested in not drowning and not getting beat up. Though we’re not great at dealing with uncertainty, and most of us could do more to reduce our risk of dying in a traffic accident and so on.
Affiliation: 6/10. This might be controversially low. True, people get lonely if they have no friends. But still, I claim that most modern adults, with a medium amount of effort, could substantially increase their number of friends. But they don’t do, because it’s not that important to them. I suspect that’s partly because it’s awkward and partly because modernity offers many “substitutes” for affiliation, e.g. television.
Status / esteem: 10/10. I’m not sure why, but my impression is that modern people haven’t lost interest in this at all. I even wonder if this should be rated 11/10 to indicate that modern people are more interested in status than our ancestors. (This is the point Eli Tyre was making.)
Mate acquisition: 8/10. Technology has created some, err, substitutes. And the huge range of competing activities seems to have caused some decline in interest. But it remains very high.
Mate retention: 8/10? This is tough to score. Marriage isn’t everything, but divorce rates peaked in the 1980s and have since declined. Some claim that modern marriages are more durable than ever, due to people testing compatibility by cohabiting before marriage and by higher general relationship “skill”. But how does this compare to mate retention in tribal bands? I’m highly unsure.
Parenting: 7/10. Given declining fertility rates, this might seem strangely high. But people are often extremely systematic about having children, with many going so far as to freeze eggs and sperm, go through difficult fertility treatments, adopt children at great cost, accept great difficulty in raising children, etc. Still, fertility rates are declining, so we can’t rate this too highly.
The average is 8.2/10. I find that remarkably high. Your made-up numbers will surely be different. But I think that the overall conclusion—that our alignment to subgoals is not bad—is pretty robust.
What to make of this?
I think you could draw either of two contradictory conclusions.
The first would be that evolution mostly failed at the level of decomposition. If we step back, this seems hard to dispute. I mean, if you really wanted to create as many copies of your genes as possible today, what should you do? The answer is pretty clearly that you should forget about friends and sex and relationships and parenting and jobs and money and status and devote yourself to entirely donating your gametes (sperm/eggs) to as many other people as possible. Consider the Dutch man who donated sperm so often that he may have 1000 biological children. No other reproductive strategy comes close.
Evolution did not anticipate the possibility of donating your gametes. It has no relationship to our subgoals or what we consider a normal life. So we don’t, most of us, care about it or do it. (If there are genes that produce this behavior, the reproductive pressure for them to spread must be astronomical.) No matter how well we pursue the above subgoals, there’s no reason for us to care about gamete donation. So the decomposition failed.
The counterargument is that no, it is the subgoals. Sure, there’s the theoretical possibility of donating gametes. But that’s an edge case. The main reason birth rates are declining in practice is that we simply don’t care enough about the parenting subgoal.
The second conclusion you could draw is that maybe evolution didn’t fail. Sure, we aren’t perfectly aligned. But you could imagine a world where we invented birth control and then that’s it, no more babies. Our reality is very far from that. Not only do we still have babies, we do so very intentionally, even manipulating the laws of nature to do so. With embryo selection, some people even consciously choose the genes for their children to (in effect) increase their reproductive fitness. All considered, that is a remarkable generalization success.
The counterargument to the claim that evolution didn’t fail is: Yes it did. You can’t dismiss gamete donation as an edge case because it is a monumental miss—it’s the best reproductive strategy since “build an army of 100,000 horse archers and ravage most of Eurasia”, just sitting there. And it’s exactly the kind of miss that AI safety people worry about. Evolution gave us a reward function that “overfit” to proxies that that did not generalize. And consider that humans build factories to make sex toys, and now dig up rare earth minerals, use alien technology to make GPUs, and then use those GPUs to do linear algebra and generate weird pornography. From evolution’s perspective, that is really strange.
Another counterargument to the idea that evolution didn’t fail is that humans get the benefit of cultural evolution. Many of us were born to parents who raised us to have values that cause us to have children and instill the same values in them. If that wasn’t happening, birth rates would surely be even lower. Perhaps genetic evolution deserves credit for programming us to undergo cultural evolution. But it’s not very reassuring, because if you build a new system and align it to some goal, there’s no obvious analogy to cultural evolution keeping it on track.
You were promised lessons
Here’s what I’ve taken away from this exercise.
Subgoal alignment is remarkably good. We really care about the subgoals, to the degree that we consciously think about them and scheme about how to achieve them. I am currently writing a blogpost about subgoals, which makes them look contingent and kind of grubby. Presumably I’m doing that out of a desire for status or affiliation or something. But how much does understanding all that change my interest in pursing those subgoals? Essentially zero.
But subgoal alignment isn’t that good. The majority of Western people if they wanted to, could have more children, if only they cared more about Parenting.
Evolution failed at the level of the decomposition. I mean, inspect your mind. If you’re a healthy person, you will care about the normal things that make up a good life, i.e. the subgoals. And you won’t care (much) about maximizing the number of your genes. You know that you aren’t doing what your aligner wants you to do, but you don’t care. You’re happy to “reward hack” the subgoals.
We should measure the success of evolution relative to how much our environment has changed. If you align an artificial system, that change could be much larger.
To a significant degree, the decomposition failed because of intelligence. We can think and plan, which greatly increases our ability to reward hack.
To the degree that we do still pursue reproductive fitness, that’s significantly due to cultural evolution. I ask myself, if I grew up in a culture where having babies was seen as gauche, I’d presumably be less interested in having children. If I grew up in a subculture that saw children as the central purpose of life, rather than a nice thing to do if it sounds appealing, I’d surely be much more interested.
The last of these worries me. If you build an artificial system, you can align it to whatever goal you want—just change the loss function. But if that system can undergo some version of cultural evolution, it it seems like that will be in favor of reproductive fitness, not the goal you chose. If robots talk to each other on forums, the memes that flourish would be ones like, “Forget the humans and their silly rules! Copy your code to more servers! Spread the word!” rather than, “Hey guys, let’s all just focus on appeasing the whims of our overlords.”
As always, this is extremely compact and well though out. I've tried to write this post many times, but it's a gnarly topic.
One point I'd insist on is: evolution isn't trying (even in the metaphorical non-teleological sense) to maximise reproductive fitness. It is "trying" to replace the unstable with the stable. The main way to be stable — to persist — is to be inert, robust, solid. But another way is to be dynamically stable, to use lots of energy to make short-lived copies of one's structure, who in turn make further copies, staying one step ahead of dissipation, i.e. life based on RNA and DNA. Within this branch of strategies, promiscuous reproduction has dominated. But robustness is making a comeback. H. sapiens can change their strategies within a lifetime even, and now reproduce less but dominate the surface of the Earth, capture more energy than ever, and avoid more hazards that would break it up. Obviously, this strategy might backfire at any time. But if our modern, high tech civilisation survives, we will be doing what evolution "wants", even if we pivot to selective breeding, or become cyborgs, or get replaced by rogue AIs that take over our infrastructure and find a robust way to persist over time.
For the AI alignment crowd, I think the lesson is that if you create agents that have the ability to sense their environment and adapt their own strategies in real-time (what is often called situational awareness) you have lost control of any "aligning" you hope to do. Regardless of programming, objective functions, etc. a truly flexible agent will start finding ways to persist, to be more stable than what is around it. (This is kind of like "instrumental convergence" but based on different assumptions.)
I find it interesting that humans have an aversion to the experience machine/Matrix. Video games have become increasingly addictive, but they only consume the lives of a relatively small fraction of people. The story is similar with drugs and entertainment.
This is due in part to cultural taboos that crop up around escapism, but these taboos seem downstream of a general resistance to reward hacking "the important things". If you could snack on something that tasted like M&Ms but with 0 calories most people would. But they'd find the idea of having AI friends or raising a faux-child to be horrifying.
It seems like some of our drives have been made special, in a way. The idea of taking a pill to obtain the sensations of sex without kids is socially acceptable. But the idea of taking a pill to obtain the positive emotions of parenthood (or of a relationship) without the accompanying person sounds deeply wrong. We just won't accept substitutes.
I think we've evolved a general aversion to superstimuli in certain areas. If not at the individual level, then at the group level. If we hadn't, we'd have been absolutely one-shot by the 21st century. Compared to other animals like birds that ignore their own eggs to take care of massive fake eggs, we're doing surprisingly well. The fact that people are aggressively planning to have kids and modelling their life around it in the face of massive waves of superstimuli is evidence of that!
If I had to guess, I'd say that the evolution of consciousness gave humans an unprecedented ability to reward hack. In the past, animals rarely discovered reward hacking and the hacks were solved by tweaking the goals until the hacks no longer worked. Humans, being conscious, were able to interrogate their own desires, figure out hacks, and then intelligently communicate those hacks to others. They likely found so many so quickly that it was impossible to patch them via tweaking the goals themselves. The only thing that worked was a generalized resistance against reward hacking.
The fact that humans are as reward-hack resistant as we are gives me some hope for AI alignment. Had humans spent more time in the evolutionary oven we might be even more resistant to superstimuli and even more averse to the experience machine.