Sadly, if you simply iterate this game by a half a decade you can see where this leads. When social media is awash with AI-generated lies, when you can no longer tell what's real and what's generated, when the video you're watching of the latest self-inflicted horror of modernity could equally be a true expose or AI slop, the power of social media as a decentralised news medium will be completely destroyed.
We've come out of an era where the centralised state media lies to you all the time, and social media has been used as a mechanism to expose these lies. Once social media is just as awash with lies as the mainstream media, this time a roiling mass of AI lies pushed out by various interest groups and individuals, the people will once again be left with no choice but to turn to the mainstream media for 'truth'. The advantage the state will have will be that, although its lies are no more true, they are at least consistent and at least form a coherent narrative: they will at least have the appearance of truth.
AI will sap our need to ever develop our soul, to ever better ourselves in any way, but so too will it completely and permanently destroy truth. Every counter to every state lie in the future world can be validly met with "that counter is just AI generated".
The monumental arrogance of historical errancy such as race-swapping could prove fully justified unless someone preserves the truth. Man to his credit has tended to be able to archive reality. We also have been given the heads up this time.
Once again, I'm struck by the prescience of Frank Herbert's Dune novels; ironic, given the significance of prescience as a plot device within the story. One example is the Ixian hunter-seekers, mentioned in the later books, which bear a surprising resemblance to drones. But much more relevant to AI is the backstory of the novels, namely the Butlerian Jihad, which takes place long before the main story. This was a holy crusade against "thinking machines", which enforced a sort of techno-feudalism on the Duniverse, with one proscription above all: "Thou shalt not make a machine in the likeness of a human mind". Thus, they had advanced technology, but no computers, with specially trained humans like Mentats, Reverend Mothers, and Face Dancers filling the gap by enhancing their human potential in various ways, some of which were rather surprising.
This expansion of human ability sans computers is in stark contrast to the history leading up to the Butlerian Jihad, in which humans became dependent on machines as their natural abilities atrophied. A couple of choice quotes:
>"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed."
>"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."
Thus, humans abolished "thinking machines" (ie computers), in order to reclaim their agency. This is all very thematic, and it's strange to see it unfolding in the real world.
Quite based, but I must reiterate that the only real Dune content is that written by Frank Herbert, with an honorable mention to the Dune Encyclopedia (which he approved of, but did not consider canon; it's a mix of good & bad). Brian & Kevin are absolute hacks who have milked & tarnished the great FH's legacy by churning out potboiler fanfic.
"pouring the ale in a manner that she knows is incorrect or bows before the Higher Authority of the customer’s iPhone and submits,"
as you later note, we already arrived at this point to a degree some time back with the advent of readily available digital information. Who are you going to believe, the Scroll God or your lying eyes? The loss of agency indeed is to be lamented. The loss of correct solutions is to be feared. AI will allow epistemological closure to transition from Soma to hemlock in short order.
I regularly do work involving coding. I use Grok via a blue tick twitter account. Recently I had an elongated task. After some time into the interaction it claimed it couldn't read specific attachments I needed it to analyse. I logged into my non blue tick twitter account, on a different browser, and asked if it could read said attachments. It said it could. I have also noted it misreads numbers, has trouble identifying columns of numbers that are clearly headed. It is still very inconsistent, or at least the version we have access to is. It can behave well and be helpful on one interaction, the next day be frustratingly idiotic and erroneous.
If it is left to have responsibility for life critical tasks, it will kill people. Now there's an idea!!
If this thing is God, then we will be dealing with the devil at the same time.
I think it will have minimal effect. I think it is overhyped. I suspect AI will simply become very efficient user manuals. Handy, but relegated to so what category very quickly. Although it has interesting potential in some areas.
As for its effect on agency. I would argue it will primarily accelerate the decline of those already displaying low agency, and perhaps trigger the higher agency types to waken up their ideas and remove more technology from their lives. The working out philosophy is this, for example. You can drive everywhere, take the lift and even work from your couch. But some get it and work out to compensate. I suspect we will see a similar thing as AI does more heavy lifting. Most will accept the convenience and decline. Some will explicitly reject it and develop their minds the old fashioned way.
As for AI becoming a god to the masses? They needn't bother. Many are already enslaved to their vices; their god is dopamine.
I think the line will hover around 50%…half will happily outsource their consumptive, pointless lives to AI, the rest will return to a more hands on lifestyle, making things, raising children, reading books, thinking about the meaning of life beyond being constantly entertained.
This is way too optimistic, isn’t it? What fraction of people are capable of thinking and living deliberately when everything has been incentivized to turn people into slaves?
I think it will be a similar percentage to those who explicitly eat well and exercise. Both are comparable as they require us to assess things then push against convenience.
I would put that comfortably under one third; possibly 20 percent. It will vary by demographic.
Another "Banger" from the Poet of the North, it made me recall a principle called the Jevons Paradox, in which a Technical Advancement results in more resources being used to make it work.
With relation to what you've said, this AI or Large Language Modeling System (LLM) is not more efficient but lesser due to it needing more resources to function than say the human Brain needs.
Its rather funny to me that there are still folk who think no these 'tools' will help them when it is more likely to destroy them. Also no of it works without the Leccy Grid. ;)
My own thoughts as well. The amount of energy required to power up and maintain this dystopia is not achievable maybe, and don't get me started on the demand for copper.
My gut feeling is that, at some level, a certain degree of "rebelliousness", of questioning authority (especially when said authority is not backed up by a pointy implement or the like), is in-built into human nature, at least for a non-negligible subset of the population. Admittedly, such a tendency has not exactly been on brilliant display for Western mankind in recent years, as we disciples of Mr. Morgoth keenly remember, but here I would argue that this has been on account of the massive domestication that Western man has undergone for almost a century. I can see people blindly following AI when they are kept like those in the orbital station in Wall-E. I strongly doubt they will be equally obedient when there is a daily struggle in a cyberpunkesque dystopia, which has been strikingly described by Mr. Morgoth several times. It does not appear to me at all likely that we will return to the material abundance that the boomers "enjoyed".
I would just say, buy hard copies of books, get rid of your TV. Go for walks and leave your phone at home. Dare yourself to talk to one new person per day (whyte person obviously). Build real networks in the real world.
P.S. I binge watched Game of Thrones with my wife recently (she's French and has never seen it) and I remember Morgoth and Woes (I think) saying that it wasn't timeless like Lord of The Rings because it was amoral and not transcendant. After the last episode's closing credits started rolling I completely underatood what they meant, it hadn't stood the test of time and was quite forgettable as a series (save a couple of outstanding episodes).
Well the problem is GRR Martin never finished the GOT books. So there's a lot of speculation as to why. In the show they just had to make it up once they caught up with him.
Yeah fair point. I wonder how he would have finished it, considering that it's a leftie critique of a more-wholesome European pagan/Christian tradition. I don't know if I want to invest the time in reading the books and working it out for myself.
Just now, the wife and I are watching "Outlander". As a Scot, I feel more attached to the characters and setting, it's very moving and the backdrops are beautiful. The English get a rough deal in how they're potrayed. Worth a watch though.
I've seen what pocket calculators & then word processors did to intelligence. I guess the future is imbeciles sitting around in nappies with a VR headset on.
Walley, the Pixar cartoon with fat people being pushed through life in bathtub shaped strollers while they sipped milkshakes predicted it all. I never understood why people weren’t more terrified by that,
Me too. They chuckled, then reached for the remote. It was an amazing dig at where the West was going. They didn't shy away either. That scene where they showed the sequence of captains over the years, each one fatter than the last, was similar.
We might be saved from this outsourcing of our thinking by the one thing that can out-compete the path of least resistance - status-seeking. It’s a bit “low class” to have to pull out your phone to do rudimentary maths vs doing it in your head, or to be completely reliant on google maps when making your way around town. When someone at work is over-reliant on GPT for their writing, it’s met with a smirk and knowing glance. That, and the fact that A.I. Is constantly wrong and doesn’t seem able to do any actual thinking (at least in my experience).. Those who really do try to outsource their thinking to it will be very obvious to spot and looked down upon by most.
The more we experience AI the less impressive it becomes. When it was new everything it spat out seemed amazing but, as time goes on, seeing the joins becomes easy. The world is so complex that its averaged out knowledge often falls short in any particular situation. Outside simplistic tasks there is rarely one right way of doing anything, as your bean trench demonstrates.
AI has been hyped to the moon by those who have pumped billions into it. I'm sure there is much hand rubbing in elite circles over its potential to enslave the masses. But it will fail and like the internet itself become a two-edged sword that can be employed to our advantage.
Very interesting and thought-provoking article, Morgoth. When you were thinking as to how someone could justify doing things the "wrong" way after ChatGPT told them otherwise, it reminded me of one of the concepts in the movie Gattaca. The characters discuss how, with the ubiquity of genetic engineering, it is considered "selfish" or "backward" to let your child be born naturally. Imagine you could, for a few extra thousand at the doctor's office, ensure your son has an extra three inches in height, or won't have any allergies, or won't be color blind. Why wouldn't you? What kind of a religious fundamentalist would object? But then, what a freak of nature we'd all become, and isn't the point of beauty it's rarity...
Almost none of the people we love in this world would exist according to the new criteria for perfection, assuming we aren’t all associating with Nobel prize winning supermodels. My son adopted from Guatemala who is the kindest, most empathetic person we have ever known, has expressive and receptive speech disorders from vaccine damages, a below average IQ, is 5’ tall, and had to have a reconstructed hip socket as a toddler. Totally wouldn’t make the cut. He is also an Eagle Scout with excellent leadership skills, a rock climbing instructor, and licensed electrician.
I’ve passed autism along with a rare neuromuscular disease on to our biological children. My husband has a severe underbite and would never have been a bathing suit model. Our birth children have high IQs but less stellar personalities. Until the advent of the computer people managed to find each other, raise families, and feel fairly confident that their children with the usual mixed bag of gifts and deficits would do the same. Suddenly normal life has become much too hard for normal people. That your judgement could deteriorate to the point where AI advice was preferred over books, human experience, instinctive feelings, old people who spent their lives perfecting skills that they hoped to pass down.
The poverty of the life you describe here is scarcely worth living. What is the value in being a passive consumer of disembodied facts?
Phenomenal piece, as always.
Sadly, if you simply iterate this game by a half a decade you can see where this leads. When social media is awash with AI-generated lies, when you can no longer tell what's real and what's generated, when the video you're watching of the latest self-inflicted horror of modernity could equally be a true expose or AI slop, the power of social media as a decentralised news medium will be completely destroyed.
We've come out of an era where the centralised state media lies to you all the time, and social media has been used as a mechanism to expose these lies. Once social media is just as awash with lies as the mainstream media, this time a roiling mass of AI lies pushed out by various interest groups and individuals, the people will once again be left with no choice but to turn to the mainstream media for 'truth'. The advantage the state will have will be that, although its lies are no more true, they are at least consistent and at least form a coherent narrative: they will at least have the appearance of truth.
AI will sap our need to ever develop our soul, to ever better ourselves in any way, but so too will it completely and permanently destroy truth. Every counter to every state lie in the future world can be validly met with "that counter is just AI generated".
And I think we'll need digital ID just get online to verify that we are not AI ourselves.
Thanks.
Hideous. You're probably right. I wonder what emergency will be forced to finally get that over the line.
Possibly the Online Harms Bill, but we shall see.
Possibly your most existential and pressing piece I've read.
So much for 2001 A Space Odyssey.
The best we can hope for is Idiocracy.
To make it worse, we know what sort of values and 20th century historical narrative our AI overlords will imprint on us.
Every single day.
The monumental arrogance of historical errancy such as race-swapping could prove fully justified unless someone preserves the truth. Man to his credit has tended to be able to archive reality. We also have been given the heads up this time.
Once again, I'm struck by the prescience of Frank Herbert's Dune novels; ironic, given the significance of prescience as a plot device within the story. One example is the Ixian hunter-seekers, mentioned in the later books, which bear a surprising resemblance to drones. But much more relevant to AI is the backstory of the novels, namely the Butlerian Jihad, which takes place long before the main story. This was a holy crusade against "thinking machines", which enforced a sort of techno-feudalism on the Duniverse, with one proscription above all: "Thou shalt not make a machine in the likeness of a human mind". Thus, they had advanced technology, but no computers, with specially trained humans like Mentats, Reverend Mothers, and Face Dancers filling the gap by enhancing their human potential in various ways, some of which were rather surprising.
This expansion of human ability sans computers is in stark contrast to the history leading up to the Butlerian Jihad, in which humans became dependent on machines as their natural abilities atrophied. A couple of choice quotes:
>"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed."
>"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them."
Thus, humans abolished "thinking machines" (ie computers), in order to reclaim their agency. This is all very thematic, and it's strange to see it unfolding in the real world.
I did a video on the Butlerian Jihad a couple of years back.
https://www.youtube.com/watch?v=iH7MyGXhlF0
Quite based, but I must reiterate that the only real Dune content is that written by Frank Herbert, with an honorable mention to the Dune Encyclopedia (which he approved of, but did not consider canon; it's a mix of good & bad). Brian & Kevin are absolute hacks who have milked & tarnished the great FH's legacy by churning out potboiler fanfic.
Magnificent. As for:
"pouring the ale in a manner that she knows is incorrect or bows before the Higher Authority of the customer’s iPhone and submits,"
as you later note, we already arrived at this point to a degree some time back with the advent of readily available digital information. Who are you going to believe, the Scroll God or your lying eyes? The loss of agency indeed is to be lamented. The loss of correct solutions is to be feared. AI will allow epistemological closure to transition from Soma to hemlock in short order.
Again, brilliant, man.
Thanks Jack.
As if agency is becoming / has become optional. Then again, maybe that was always the case.
I regularly do work involving coding. I use Grok via a blue tick twitter account. Recently I had an elongated task. After some time into the interaction it claimed it couldn't read specific attachments I needed it to analyse. I logged into my non blue tick twitter account, on a different browser, and asked if it could read said attachments. It said it could. I have also noted it misreads numbers, has trouble identifying columns of numbers that are clearly headed. It is still very inconsistent, or at least the version we have access to is. It can behave well and be helpful on one interaction, the next day be frustratingly idiotic and erroneous.
If it is left to have responsibility for life critical tasks, it will kill people. Now there's an idea!!
If this thing is God, then we will be dealing with the devil at the same time.
I think it will have minimal effect. I think it is overhyped. I suspect AI will simply become very efficient user manuals. Handy, but relegated to so what category very quickly. Although it has interesting potential in some areas.
As for its effect on agency. I would argue it will primarily accelerate the decline of those already displaying low agency, and perhaps trigger the higher agency types to waken up their ideas and remove more technology from their lives. The working out philosophy is this, for example. You can drive everywhere, take the lift and even work from your couch. But some get it and work out to compensate. I suspect we will see a similar thing as AI does more heavy lifting. Most will accept the convenience and decline. Some will explicitly reject it and develop their minds the old fashioned way.
As for AI becoming a god to the masses? They needn't bother. Many are already enslaved to their vices; their god is dopamine.
" perhaps trigger the higher agency types to waken up their ideas and remove more technology from their lives"
That's a best-case scenario. I do think truth is likely to be maintained in the contemporary versions of monasteries.
I think the line will hover around 50%…half will happily outsource their consumptive, pointless lives to AI, the rest will return to a more hands on lifestyle, making things, raising children, reading books, thinking about the meaning of life beyond being constantly entertained.
This is way too optimistic, isn’t it? What fraction of people are capable of thinking and living deliberately when everything has been incentivized to turn people into slaves?
I think it will be a similar percentage to those who explicitly eat well and exercise. Both are comparable as they require us to assess things then push against convenience.
I would put that comfortably under one third; possibly 20 percent. It will vary by demographic.
Another "Banger" from the Poet of the North, it made me recall a principle called the Jevons Paradox, in which a Technical Advancement results in more resources being used to make it work.
With relation to what you've said, this AI or Large Language Modeling System (LLM) is not more efficient but lesser due to it needing more resources to function than say the human Brain needs.
Its rather funny to me that there are still folk who think no these 'tools' will help them when it is more likely to destroy them. Also no of it works without the Leccy Grid. ;)
Blessings to you and yours Morgoth.
My own thoughts as well. The amount of energy required to power up and maintain this dystopia is not achievable maybe, and don't get me started on the demand for copper.
Thanks.
Christ, it is all so depressing. Great writing, Morgoth, as ever. Unless you've been replaced by an iMorgoth or a Smart Morgoth.....
Lol. But don't worry, the machines will never be able to replicate our accents properly. Never!
My gut feeling is that, at some level, a certain degree of "rebelliousness", of questioning authority (especially when said authority is not backed up by a pointy implement or the like), is in-built into human nature, at least for a non-negligible subset of the population. Admittedly, such a tendency has not exactly been on brilliant display for Western mankind in recent years, as we disciples of Mr. Morgoth keenly remember, but here I would argue that this has been on account of the massive domestication that Western man has undergone for almost a century. I can see people blindly following AI when they are kept like those in the orbital station in Wall-E. I strongly doubt they will be equally obedient when there is a daily struggle in a cyberpunkesque dystopia, which has been strikingly described by Mr. Morgoth several times. It does not appear to me at all likely that we will return to the material abundance that the boomers "enjoyed".
I would just say, buy hard copies of books, get rid of your TV. Go for walks and leave your phone at home. Dare yourself to talk to one new person per day (whyte person obviously). Build real networks in the real world.
P.S. I binge watched Game of Thrones with my wife recently (she's French and has never seen it) and I remember Morgoth and Woes (I think) saying that it wasn't timeless like Lord of The Rings because it was amoral and not transcendant. After the last episode's closing credits started rolling I completely underatood what they meant, it hadn't stood the test of time and was quite forgettable as a series (save a couple of outstanding episodes).
Anyway, I'm blethering. Thanks again Morgoth.
Well the problem is GRR Martin never finished the GOT books. So there's a lot of speculation as to why. In the show they just had to make it up once they caught up with him.
Yeah fair point. I wonder how he would have finished it, considering that it's a leftie critique of a more-wholesome European pagan/Christian tradition. I don't know if I want to invest the time in reading the books and working it out for myself.
Just now, the wife and I are watching "Outlander". As a Scot, I feel more attached to the characters and setting, it's very moving and the backdrops are beautiful. The English get a rough deal in how they're potrayed. Worth a watch though.
I've seen what pocket calculators & then word processors did to intelligence. I guess the future is imbeciles sitting around in nappies with a VR headset on.
Thank Fuck I'll be dead.
That is quite an image.
Walley, the Pixar cartoon with fat people being pushed through life in bathtub shaped strollers while they sipped milkshakes predicted it all. I never understood why people weren’t more terrified by that,
Me too. They chuckled, then reached for the remote. It was an amazing dig at where the West was going. They didn't shy away either. That scene where they showed the sequence of captains over the years, each one fatter than the last, was similar.
🤣 You've seen "Idiocracy"? If not I recommend it. It's looking more and more likely all the time, scarily so.
Indeed. A little too close to the bone at times.
We might be saved from this outsourcing of our thinking by the one thing that can out-compete the path of least resistance - status-seeking. It’s a bit “low class” to have to pull out your phone to do rudimentary maths vs doing it in your head, or to be completely reliant on google maps when making your way around town. When someone at work is over-reliant on GPT for their writing, it’s met with a smirk and knowing glance. That, and the fact that A.I. Is constantly wrong and doesn’t seem able to do any actual thinking (at least in my experience).. Those who really do try to outsource their thinking to it will be very obvious to spot and looked down upon by most.
How long before it becomes Grok am I real?
Great work!
The more we experience AI the less impressive it becomes. When it was new everything it spat out seemed amazing but, as time goes on, seeing the joins becomes easy. The world is so complex that its averaged out knowledge often falls short in any particular situation. Outside simplistic tasks there is rarely one right way of doing anything, as your bean trench demonstrates.
AI has been hyped to the moon by those who have pumped billions into it. I'm sure there is much hand rubbing in elite circles over its potential to enslave the masses. But it will fail and like the internet itself become a two-edged sword that can be employed to our advantage.
Very interesting and thought-provoking article, Morgoth. When you were thinking as to how someone could justify doing things the "wrong" way after ChatGPT told them otherwise, it reminded me of one of the concepts in the movie Gattaca. The characters discuss how, with the ubiquity of genetic engineering, it is considered "selfish" or "backward" to let your child be born naturally. Imagine you could, for a few extra thousand at the doctor's office, ensure your son has an extra three inches in height, or won't have any allergies, or won't be color blind. Why wouldn't you? What kind of a religious fundamentalist would object? But then, what a freak of nature we'd all become, and isn't the point of beauty it's rarity...
Yes, the true horror is realizing that the progressive side in this have perfectly sound arguments. Dentistry being an old classic.
Almost none of the people we love in this world would exist according to the new criteria for perfection, assuming we aren’t all associating with Nobel prize winning supermodels. My son adopted from Guatemala who is the kindest, most empathetic person we have ever known, has expressive and receptive speech disorders from vaccine damages, a below average IQ, is 5’ tall, and had to have a reconstructed hip socket as a toddler. Totally wouldn’t make the cut. He is also an Eagle Scout with excellent leadership skills, a rock climbing instructor, and licensed electrician.
I’ve passed autism along with a rare neuromuscular disease on to our biological children. My husband has a severe underbite and would never have been a bathing suit model. Our birth children have high IQs but less stellar personalities. Until the advent of the computer people managed to find each other, raise families, and feel fairly confident that their children with the usual mixed bag of gifts and deficits would do the same. Suddenly normal life has become much too hard for normal people. That your judgement could deteriorate to the point where AI advice was preferred over books, human experience, instinctive feelings, old people who spent their lives perfecting skills that they hoped to pass down.
The poverty of the life you describe here is scarcely worth living. What is the value in being a passive consumer of disembodied facts?
Great piece. I'm reminded of TS Eliot's lines from 'The Rock':
"Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?"
We've been steadily losing deeper coherence for decades now; as we lose the instincts of significance, the sludge of raw data grows.