Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"
Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".
Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".
Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".
Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
> [at the hearing regarding the civil rights of androids like Data]
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.
Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
It is in fact more complicated than most people assume.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
It's not a person. Glad we were able to get this resolved so quickly.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.
You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.
Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm.
Why would an ai with a mind remove liability from the company?
why would an ai without a mind remove liability from the company?
In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.
In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world.
It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.
The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.
We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.
>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
The current implementation as stateless matrix multiplication... yeah there's nothing going on there.
> If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?
Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?
Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).
It's not just memory, the physical body also encodes trauma and stress (The Body Keeps the Score is a book that touches on this).
I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.
Perhaps off-topic, but there's some people who believe that this can be what happens during surgery under anesthesia, if the dosage is off.
From what I understand, there's 3 parts to modern anesthesia: Blocking pain signals, preventing the formation of memories, and inducing paralysis of the major muscle groups. If the former is dialed in too weakly, it's possible that the patient is feeling pain, unable to do anything about it, but won't remember it at all when they wake up.
(Looking it up now, it seems that making the patient unconscious is perhaps the same drug that prevents memory formation... I realize now I understand this less well than I thought I did.)
They don’t have persistent memory though. Models are static, what you have is a model that will reprocess the whole series of prompts + some data fetched from a data store. A harness is just a while loop prompting an LLM and executing tool calls, but the model is purely static
The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.
It's different, but it is not at all clear whether or not that matters for the purposes of whether or not they can experience things, seeing as we don't know how to measure that even for other humans other than by relying on self-reporting to infer without other evidence that their experience matches ours.
We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience.
Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.
Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.
What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?
After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for dry cleaners, your brain kinda disconnects from reality and just assumes that it is correct about everything.
He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"
I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).
Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.
does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?
Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways.
IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?
Whenever I see comments like this it just reminds me how ignorant people are of neurobiology.
The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.
A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.
It's always interesting since my train of thought always goes down:
- Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations
- Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?
- I have no clue if they think or feel or ....
Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute
Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.
You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.
I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.
Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
>The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
Bollocks.
If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".
We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI?
(I don't think AI is conscious, just following the argument.)
This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.
It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs.
Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
That's the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There's a statement that "consciousness is very likely biological"... and then that's it. No argument for it, no citation.
If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."
I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.
Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.
Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is.
So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.
It's nice when you can set an impossibly high bar before which you can dismiss all your personal responsibility.
Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?
Why is this only ever demanded of people who criticize the premise that LLMs are conscious beings, whereas the people who believe it simply have to gesture vaguely in the direction of some correlation in language between LLMs and humans, and the matter is considered closed?
What is the empirically tested basis for the null hypothesis being that LLMs are conscious until proven otherwise?
There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something else (we have no clue).
It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias.
Anthropomorphization? Completely disregarding the possibility of consciousness is no better.
Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!
Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being "conscious" isn't really the moral line in the sand many people are drawing in response to this article.
Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
And most people wouldn't eat cats, but would eat some of pig, cattle, chicken, what's the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn't eat any animals. But we clearly have a way of picking and choosing which is OK and which isn't. And in both cases we generally don't give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.
I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.
That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.
If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.
I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.
None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!
I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights.
I don't know if next door's pet dog is either, but that has animal rights.
Perhaps then the answer is simply, show some respect.
Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.
If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.
Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).
This consideration should be case by case for AI too.
I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed:
“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”
It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.
I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web.
In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.
The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there
Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won't be able to tell me what you had for breakfast 26 days ago or would only know if it's the same thing you have every day. Does that make you lack consciousness?
Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don't you think you would still be conscious but back to a previous state?
If you rewind my brain I expect to give you a similar answer to the one I gave you before. Which isn’t the case for LLM. You can literally replay a positive answer, then get a negative response that isn’t consistent at all with the one it previously generated.
I'd argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism).
In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don't think those ones would differ so much from the "random ones" to call the firsts conscious and the second non-conscious.
I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.
I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious.
Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.
I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.
Everyone is (predictably) getting distracted by the consciousness claims.
The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.
TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it.
Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.
I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?
But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.
Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".
Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".
Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".
Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
I mean it's WITH AI!
You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.
Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
Why would an ai with a mind remove liability from the company? why would an ai without a mind remove liability from the company?
In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.
It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.
The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.
We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.
Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
The current implementation as stateless matrix multiplication... yeah there's nothing going on there.
They do, during training
Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).
I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.
From what I understand, there's 3 parts to modern anesthesia: Blocking pain signals, preventing the formation of memories, and inducing paralysis of the major muscle groups. If the former is dialed in too weakly, it's possible that the patient is feeling pain, unable to do anything about it, but won't remember it at all when they wake up.
(Looking it up now, it seems that making the patient unconscious is perhaps the same drug that prevents memory formation... I realize now I understand this less well than I thought I did.)
It's also a rhetorical move I've seen several times here...
Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.
He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"
The key variable people consider is: Can this thing harm me back?
Seems as simple as "will this action bring social shame/criticism" for any individual decision.
IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?
The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.
If you firmly believe this to be true, then you should stop using LLMs.
- Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations
- Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?
- I have no clue if they think or feel or ....
Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute
Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.
I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.
Bollocks.
If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".
(I don't think AI is conscious, just following the argument.)
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."
I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.
Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?
What is the empirically tested basis for the null hypothesis being that LLMs are conscious until proven otherwise?
It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias.
Anthropomorphization? Completely disregarding the possibility of consciousness is no better.
Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!
Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.
I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.
None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!
I don't know if next door's pet dog is either, but that has animal rights.
Perhaps then the answer is simply, show some respect.
Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.
If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.
Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).
This consideration should be case by case for AI too.
“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”
It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.
He’s still just a character.
In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.
Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB.
Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.
But connect them all together...
>They do not have innate preferences or underlying motivations
Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.
Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.
They're trained on human behavior. Whether or not they genuinely have feelings - they sure as heck behave as though they do.
And when you treat people well, they do better work for you.
No brainer.
You want to enjoy having an AI slave do your "work" for you forever? Have fun. I'm not reading this reinvent-dualism-from-apple-sauce slop.
The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.
Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.