Framing the harder alignment problem as succession is clarifying and a great intuition pump.
Looking forward to value mysticism. I wouldn’t have said this before I spent serious time meditating, but I strongly believe value is not something that is fixed - you can’t just specify the goals and constraints and compute a solution to it.
(Subjective perception of) value , seems to be dynamic, unstable, and constantly renegotiated - as well as have a mysterious relationship to consciousness / subjective awareness. As long as things change, which they always do, values can never be fixed, and there can thus never be a single solution to satisfying them.
I also like this quote from Ken Wilbur: “the subject of knowing co-creates the reality it knows, and that therefore some aspects of reality will literally be created by the subject and the interpretation it gives to that reality.”
Also, the Buddha spoke of these ideas millennia ago.
My current take here is that for the purposes of succession, you actually don't need to solve moral philosophy in full, forever, mostly because of the possibility of moral trade/compromise both causally and acausally that lets most people get most of what they want while only meeting the first bar.
(To be clear, I'm not a successionist in the sense that I don't think it's acceptable to intentionally increase the probability we all die because you think human extinction is good/fine, but of course all of the debates is between non-successionists in the way I defined it.)
While I don't think liberalism or democracy will survive AI progress, I do think that it's (maybe) tractable, and important/neglected to port over one of the concepts/founding reasons for liberalism and democracy, which is attempting to compromise/trade across people with vastly different moral views.
Or in the words of John Rawls:
The problem of political liberalism is: How is it possible that there may exist over time a stable and just society of free and equal citizens profoundly divided by reasonable though incompatible religious, philosophical, and moral doctrines? Put another way: How is it possible that deeply opposed though reasonable comprehensive doctrines may live together and all affirm the political conception of a constitutional regime? What is the structure and content of a political conception that can gain the support of such an overlapping consensus? These are among the questions that political liberalism tries to answer.
And the answer for the future is trade/compromise can replace voting/electoral systems.
This still requires a lot of improvement in the political situation, such that people can consider trade/compromise to be worth it rather than their easy reaction of banning/trying to inflict violence on opposing groups, but contra your essay, I don't believe the succession problem is as hard as you state.
(Some other reasons why the succession problem is easy, conditional on having corrigible AIs is because unlike humans, who grow old and die, corrigible AIs will be living for a very, very long time, and their only limit is how much free energy they can get/generate, and once they die, the universe is maximally boring and in thermodynamic equlibrium, where no more work is possible, or the decay of the vacuum happened and killed everyone, and not being able to live forever is part of my explanation for why dictatorships eventually trend towards the same bad endpoint from a citizen perspective, and relatedly it's much easier to lock-in desirable values post-AGI.)
Framing the harder alignment problem as succession is clarifying and a great intuition pump.
Looking forward to value mysticism. I wouldn’t have said this before I spent serious time meditating, but I strongly believe value is not something that is fixed - you can’t just specify the goals and constraints and compute a solution to it.
(Subjective perception of) value , seems to be dynamic, unstable, and constantly renegotiated - as well as have a mysterious relationship to consciousness / subjective awareness. As long as things change, which they always do, values can never be fixed, and there can thus never be a single solution to satisfying them.
A relevant contemporary thinker is David Chapman: https://meaningness.com
I also like this quote from Ken Wilbur: “the subject of knowing co-creates the reality it knows, and that therefore some aspects of reality will literally be created by the subject and the interpretation it gives to that reality.”
Also, the Buddha spoke of these ideas millennia ago.
Meditation genuinely feels like a fast path to grappling with the complexity of value.
(Crossposted from LessWrong, with the links now being added in the end instead of in the words themselves. Link below for the original post):
https://www.lesswrong.com/posts/7dkasKLC7abXhn9JZ?commentId=8f7gLqyYq9Cps2TZe
My current take here is that for the purposes of succession, you actually don't need to solve moral philosophy in full, forever, mostly because of the possibility of moral trade/compromise both causally and acausally that lets most people get most of what they want while only meeting the first bar.
(To be clear, I'm not a successionist in the sense that I don't think it's acceptable to intentionally increase the probability we all die because you think human extinction is good/fine, but of course all of the debates is between non-successionists in the way I defined it.)
While I don't think liberalism or democracy will survive AI progress, I do think that it's (maybe) tractable, and important/neglected to port over one of the concepts/founding reasons for liberalism and democracy, which is attempting to compromise/trade across people with vastly different moral views.
Or in the words of John Rawls:
The problem of political liberalism is: How is it possible that there may exist over time a stable and just society of free and equal citizens profoundly divided by reasonable though incompatible religious, philosophical, and moral doctrines? Put another way: How is it possible that deeply opposed though reasonable comprehensive doctrines may live together and all affirm the political conception of a constitutional regime? What is the structure and content of a political conception that can gain the support of such an overlapping consensus? These are among the questions that political liberalism tries to answer.
And the answer for the future is trade/compromise can replace voting/electoral systems.
This still requires a lot of improvement in the political situation, such that people can consider trade/compromise to be worth it rather than their easy reaction of banning/trying to inflict violence on opposing groups, but contra your essay, I don't believe the succession problem is as hard as you state.
(Some other reasons why the succession problem is easy, conditional on having corrigible AIs is because unlike humans, who grow old and die, corrigible AIs will be living for a very, very long time, and their only limit is how much free energy they can get/generate, and once they die, the universe is maximally boring and in thermodynamic equlibrium, where no more work is possible, or the decay of the vacuum happened and killed everyone, and not being able to live forever is part of my explanation for why dictatorships eventually trend towards the same bad endpoint from a citizen perspective, and relatedly it's much easier to lock-in desirable values post-AGI.)
Some important links to read down below:
https://www.forethought.org/research/convergence-and-compromise#31-trade-and-compromise
https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future#scenario-1-causal-coordination
https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future#scenario-2-ecl