I am so tired. This is very well written but it made me realize I can't think about this shit anymore. I just want reality to slow down for five fucking seconds so I can catch my breathe. If that makes me selfish, so be it.
A lot of good thinking and exploring in this post of yours, and I want to name what I think you've gotten right before pushing: the moral road instead of the compass (direction without a fixed yardstick), succession as locally-valid chains rather than similarity leaps, Mill's "too complex and indefinite an end" over Bentham's fluid, and above all "to avoid specification-gaming, avoid the specification" with apprenticeship as the alternative. Those four fit together more tightly than the essay lets them, and I think three moves would close the loop.
First, the road needs a way to tell a step forward from a step sideways, or "progress" quietly reintroduces the compass. You already have the material: "shown every link in the chain" — your ancestors judging us by the path, not by a destination. Make that the rule. Measure from the origin, not toward a goal: a step is progress when it keeps more of the world in view and holds together better with what came before — wider context, greater coherence. That's what separates the jellyfish reductio from the abolition of slavery: the first widens without cohering, the second widens what was already, on honest inspection, in view. The moral circle and the moral road aren't rivals; the circle is the context, the road is the coherence, and you need both.
Second, the flame-boxes test the wrong property. Your confidence about biological children rests on similarity of substrate — "built like me, so the flame is probably there." But your own ancestor example says the real test is the chain: every link a participant who inherited the valuing and revised it under contact. That criterion doesn't care what the box is made of; it cares whether the flame was transferred by the process you describe in your final section — apprenticeship, participation, tacit inheritance — or by specification. Which yields the sharpest consequence of your own argument: handing off to AIs is "a jump onto rails" only if it's done by spec. Done by apprenticeship, over many links, graded at each one, it's a chain by your own definition. The line isn't human/AI; it's specified/apprenticed.
Third, "judgment should stay with us" is right about the present locus and stronger if stated about the process: what can't be handed off is the valuing — the active, revising judgment you've described — and humans are where it currently runs. Keep it there; but the reason isn't that brains are the only possible ground. It's that nothing yet has been apprenticed long enough to be trusted with a link. That's a claim about the chain's current length, not about substrate — and it keeps your position honest when the chain gets longer.
The problem is not change, it is the rate of change. There is a rate of change threshold that nobody is happy with and AI is showing us that that threshold is close. The problem is that, to limit the rate of change to something that most humans are happy with requires a dictatorship and SGI is the only thing that could achieve that.
Phenomenal essay, these are some of the questions in AI I think are most under-discussed. I think it'd be awesome to show examples of future scenarios to better show concretely which scenarios align with "anti-succession + keeping human wills in control" and which don't. I'm imagining scenarios from Bostrom, AI 2040, sci-fi, and new variations, which encapsulate AI, genetic technology, etc. I may try writing this.
I am so tired. This is very well written but it made me realize I can't think about this shit anymore. I just want reality to slow down for five fucking seconds so I can catch my breathe. If that makes me selfish, so be it.
A lot of good thinking and exploring in this post of yours, and I want to name what I think you've gotten right before pushing: the moral road instead of the compass (direction without a fixed yardstick), succession as locally-valid chains rather than similarity leaps, Mill's "too complex and indefinite an end" over Bentham's fluid, and above all "to avoid specification-gaming, avoid the specification" with apprenticeship as the alternative. Those four fit together more tightly than the essay lets them, and I think three moves would close the loop.
First, the road needs a way to tell a step forward from a step sideways, or "progress" quietly reintroduces the compass. You already have the material: "shown every link in the chain" — your ancestors judging us by the path, not by a destination. Make that the rule. Measure from the origin, not toward a goal: a step is progress when it keeps more of the world in view and holds together better with what came before — wider context, greater coherence. That's what separates the jellyfish reductio from the abolition of slavery: the first widens without cohering, the second widens what was already, on honest inspection, in view. The moral circle and the moral road aren't rivals; the circle is the context, the road is the coherence, and you need both.
Second, the flame-boxes test the wrong property. Your confidence about biological children rests on similarity of substrate — "built like me, so the flame is probably there." But your own ancestor example says the real test is the chain: every link a participant who inherited the valuing and revised it under contact. That criterion doesn't care what the box is made of; it cares whether the flame was transferred by the process you describe in your final section — apprenticeship, participation, tacit inheritance — or by specification. Which yields the sharpest consequence of your own argument: handing off to AIs is "a jump onto rails" only if it's done by spec. Done by apprenticeship, over many links, graded at each one, it's a chain by your own definition. The line isn't human/AI; it's specified/apprenticed.
Third, "judgment should stay with us" is right about the present locus and stronger if stated about the process: what can't be handed off is the valuing — the active, revising judgment you've described — and humans are where it currently runs. Keep it there; but the reason isn't that brains are the only possible ground. It's that nothing yet has been apprenticed long enough to be trusted with a link. That's a claim about the chain's current length, not about substrate — and it keeps your position honest when the chain gets longer.
You can find more highly relevant thinking at http://arrowofmorality.org.
The problem is not change, it is the rate of change. There is a rate of change threshold that nobody is happy with and AI is showing us that that threshold is close. The problem is that, to limit the rate of change to something that most humans are happy with requires a dictatorship and SGI is the only thing that could achieve that.
Phenomenal essay, these are some of the questions in AI I think are most under-discussed. I think it'd be awesome to show examples of future scenarios to better show concretely which scenarios align with "anti-succession + keeping human wills in control" and which don't. I'm imagining scenarios from Bostrom, AI 2040, sci-fi, and new variations, which encapsulate AI, genetic technology, etc. I may try writing this.