Big fan of the idea that there is sometimes value in things even if you do not have the words or arguments to articulate it. Just because you cannot measure or explain things like feelings or experience does not mean it isn't real!
I don’t understand the choice to conflate between two sets of “successionism” here. As you point out yourself, there’s a world of difference between having decisions made on your behalf and being directly supplanted by a separate species. It's possible to imagine many very good worlds where humans exist happily under an AI handoff, while experiential successionism is basically guaranteed to annihilate human value.
I have no gripe with the argument against the experiential strains. Why flatten the great and endorsed values that humans already have into a half-baked, "objective" proxy? I do, however, think that labeling AI decision making (control) as a form of successionism is unproductive.
I think it’s clearly true, for example, that we could not trust humans with nuclear weapons indefinitely. If you leave room for top-level human decision making in place, humans are eventually going to make terrible, irreversible choices about how to use it. There’s clearly some level of decision making that needs to be delegated to perfectly rational and impartial agents in perpetuity.
I agree that the need for this kind of objective decision making is much weaker at the cultural/individual level. But there are still plenty of ways for people to make self-destructive, irreversible choices at the individual level that seem moral to guard against. Death is a straightforward example of this: many people have a deeply belief held that the end of life is what brings meaning to the rest of it. But this value only exists as the result of an instrumental coping cultural mechanism for the inevitability of death---once it's technologically obsoleted, is there no moral obligation to try and convince them otherwise? Most people reject the idea that teenagers can make an informed decision to commit suicide, and we intervene on their behalf. Adult humans are themselves children relative to the rest of their potential lifespan, so don't they warrant the same intervention?
Definitely looking forward to see where the line for these dilemmas is drawn in the next installment!
This is a great article. But I think one factor it does not talk about is just the desire to be on the winning side. The arguments that some kind of AI succession is inevitable (whether they are right or wrong) can be incredibly convincing, especially if you are already committed to certain kinds of intelligence centric world views like rationalism.
Once you genuinely become convinced that something is inevitable I think it becomes pretty psychologically unsustainable to continue to oppose it (except maybe if you have some clear concept of glorious defeat like martyrdom).
I’m all for human agency, but I think the big problem you’re not engaging with here is: What happens when some of those humans want to exercise that agency by creating morally relevant Als? Or delegate some of their decision-making? It seems likely that letting human agency rip would lead to a world that is mostly run by AIs, where most morally relevant experiences are being had by AIs. It seems quite difficult to articulate a way to avoid that future, that doesn’t involve a lot of restrictions on human agency, i.e. some degree of authoritarianism.
A lot of the “successionists” you list seem to be making a good faith effort to grapple with this question, and I don’t think it’s particularly helpful (or liberal) to frame them as representing some kind of “bad internet meme” that we need “memetic defenses” against (which I guess entails calling them “cucks”).
I do think it’s worth thinking about how to preserve as much human agency as possible in the future! And I hope future posts in this series will say more about how to go about that. But the definition of “successionism” you adopt here seems to imply that non-human agency and value are in a zero-sum relationship to human agency and value, which doesn’t seem obvious at all to me.
This is a long strawman. None of the people you represent as successionists think humanity should not continue existing and all of them agree with you that the replacement of all life with non-conscious AI is a moral catastrophe. Jan's essay explicitly attacks successionist memetics and avoids focusing on counterarguments about the foundational, objective successionist claim, which he acknowledges is valid: "Maybe some future version of humanity will want to do some handover..."
There are narrower arguments you could make that would contribute to the conversation, e.g.
- that some or all concrete successionist plans don't recognize what is valuable and potentially unique about humanity, and then say more about what those things are and encourage efforts to support their expression (your Forest succession story is also a strawman, 'plastic' is doing all the work for you there).
- that succesionist plans make unfounded claims about how fast we will develop the technologies that supposedly drive succession and are therefore distractions from better opportunities to improve realistic opportunities to improve human capacity and experience
- that conscious AI is unlikely/impossible and successionist plans will lead to giving personhood rights to AIs that don't deserve them and/or reinforce AI psychosis, sensemaking collapse, the collapse of intergenerational cultural transmission, etc.
- that successionists will prematurely claim to have created new species to justify quasi-racist class distinctions and fascistic government policies.
I hope you'll make these arguments, but none of them are against the basic claim successionists make.
>Secondly, and more fundamentally: when you look at math and feel that it has value, this gives you evidence that your brain looking at math has moral value. You cannot derive from this that math, existing outside your brain, would also have moral value. (More on this in a later post)
Unless you are going ahead with solipsism, you can reasonably extrapolate that math has value also in other minds, whether biological (human) or, with somewhat less credence, artificial.
Go far enough back and you will have an ancestor who was a fish (or similar). Would that ancestor have wanted to be replaced by you? Almost certainly not, or at least not within a "small" number of fsihy generation. On the timescale that it actually took? Maybe. It was so gradual that nobody really noticed.
The problem for us is that we can forsee ourselves being replaced in our own lifetimes, or at least those of our grandchildren. Very few people are up for that.
Best hope would seem to be that an aligned ASI would agree to "phase us out" gradually over a period of maybe 1000 years, which is probably a better offer than we'd get from augmented humans.
The idea that Homo Sapiens in 100,000 years will be almost indistinguishable from Homo Sapiens today is clearly ridiculous and could probably only happen if ASI made it so.
Many good points. Where I disagree is the ~"listen to what you actually want, Pythagoreans are somewhat wrong" is that what I want very likely is representable in math abstractions, although not the neat explicit S2 math. Harari is often dismissed in these circles, but I think he actually makes a good point about this in Homo Deus.
Big fan of the idea that there is sometimes value in things even if you do not have the words or arguments to articulate it. Just because you cannot measure or explain things like feelings or experience does not mean it isn't real!
Great article! Few thoughts on where I disagree.
I don’t understand the choice to conflate between two sets of “successionism” here. As you point out yourself, there’s a world of difference between having decisions made on your behalf and being directly supplanted by a separate species. It's possible to imagine many very good worlds where humans exist happily under an AI handoff, while experiential successionism is basically guaranteed to annihilate human value.
I have no gripe with the argument against the experiential strains. Why flatten the great and endorsed values that humans already have into a half-baked, "objective" proxy? I do, however, think that labeling AI decision making (control) as a form of successionism is unproductive.
I think it’s clearly true, for example, that we could not trust humans with nuclear weapons indefinitely. If you leave room for top-level human decision making in place, humans are eventually going to make terrible, irreversible choices about how to use it. There’s clearly some level of decision making that needs to be delegated to perfectly rational and impartial agents in perpetuity.
I agree that the need for this kind of objective decision making is much weaker at the cultural/individual level. But there are still plenty of ways for people to make self-destructive, irreversible choices at the individual level that seem moral to guard against. Death is a straightforward example of this: many people have a deeply belief held that the end of life is what brings meaning to the rest of it. But this value only exists as the result of an instrumental coping cultural mechanism for the inevitability of death---once it's technologically obsoleted, is there no moral obligation to try and convince them otherwise? Most people reject the idea that teenagers can make an informed decision to commit suicide, and we intervene on their behalf. Adult humans are themselves children relative to the rest of their potential lifespan, so don't they warrant the same intervention?
Definitely looking forward to see where the line for these dilemmas is drawn in the next installment!
Brilliant. I'm looking forward to the rest of this series.
Two recent posts this reminds me of:
* Ajeya - Loosening the grip of utilitarianism: https://acotra.substack.com/p/loosening-the-grip-of-utilitarianism
* Andy - What does it mean for AI to be democratic? https://blog.andymasley.com/p/what-does-it-mean-for-ai-to-be-democratic
As a recovering apologizer-for-my-atoms, I'm glad to see this strain of more thoughtful pluralistic thinking entering the zeitgeist.
This is a great article. But I think one factor it does not talk about is just the desire to be on the winning side. The arguments that some kind of AI succession is inevitable (whether they are right or wrong) can be incredibly convincing, especially if you are already committed to certain kinds of intelligence centric world views like rationalism.
Once you genuinely become convinced that something is inevitable I think it becomes pretty psychologically unsustainable to continue to oppose it (except maybe if you have some clear concept of glorious defeat like martyrdom).
I’m all for human agency, but I think the big problem you’re not engaging with here is: What happens when some of those humans want to exercise that agency by creating morally relevant Als? Or delegate some of their decision-making? It seems likely that letting human agency rip would lead to a world that is mostly run by AIs, where most morally relevant experiences are being had by AIs. It seems quite difficult to articulate a way to avoid that future, that doesn’t involve a lot of restrictions on human agency, i.e. some degree of authoritarianism.
A lot of the “successionists” you list seem to be making a good faith effort to grapple with this question, and I don’t think it’s particularly helpful (or liberal) to frame them as representing some kind of “bad internet meme” that we need “memetic defenses” against (which I guess entails calling them “cucks”).
I do think it’s worth thinking about how to preserve as much human agency as possible in the future! And I hope future posts in this series will say more about how to go about that. But the definition of “successionism” you adopt here seems to imply that non-human agency and value are in a zero-sum relationship to human agency and value, which doesn’t seem obvious at all to me.
This is a long strawman. None of the people you represent as successionists think humanity should not continue existing and all of them agree with you that the replacement of all life with non-conscious AI is a moral catastrophe. Jan's essay explicitly attacks successionist memetics and avoids focusing on counterarguments about the foundational, objective successionist claim, which he acknowledges is valid: "Maybe some future version of humanity will want to do some handover..."
There are narrower arguments you could make that would contribute to the conversation, e.g.
- that some or all concrete successionist plans don't recognize what is valuable and potentially unique about humanity, and then say more about what those things are and encourage efforts to support their expression (your Forest succession story is also a strawman, 'plastic' is doing all the work for you there).
- that succesionist plans make unfounded claims about how fast we will develop the technologies that supposedly drive succession and are therefore distractions from better opportunities to improve realistic opportunities to improve human capacity and experience
- that conscious AI is unlikely/impossible and successionist plans will lead to giving personhood rights to AIs that don't deserve them and/or reinforce AI psychosis, sensemaking collapse, the collapse of intergenerational cultural transmission, etc.
- that successionists will prematurely claim to have created new species to justify quasi-racist class distinctions and fascistic government policies.
I hope you'll make these arguments, but none of them are against the basic claim successionists make.
>Secondly, and more fundamentally: when you look at math and feel that it has value, this gives you evidence that your brain looking at math has moral value. You cannot derive from this that math, existing outside your brain, would also have moral value. (More on this in a later post)
Unless you are going ahead with solipsism, you can reasonably extrapolate that math has value also in other minds, whether biological (human) or, with somewhat less credence, artificial.
It's all about the rate of change.
Go far enough back and you will have an ancestor who was a fish (or similar). Would that ancestor have wanted to be replaced by you? Almost certainly not, or at least not within a "small" number of fsihy generation. On the timescale that it actually took? Maybe. It was so gradual that nobody really noticed.
The problem for us is that we can forsee ourselves being replaced in our own lifetimes, or at least those of our grandchildren. Very few people are up for that.
Best hope would seem to be that an aligned ASI would agree to "phase us out" gradually over a period of maybe 1000 years, which is probably a better offer than we'd get from augmented humans.
The idea that Homo Sapiens in 100,000 years will be almost indistinguishable from Homo Sapiens today is clearly ridiculous and could probably only happen if ASI made it so.
Many good points. Where I disagree is the ~"listen to what you actually want, Pythagoreans are somewhat wrong" is that what I want very likely is representable in math abstractions, although not the neat explicit S2 math. Harari is often dismissed in these circles, but I think he actually makes a good point about this in Homo Deus.