I work in the linguistics department at Stellar Dynamics. My job is to analyze conversations between our personal AI assistant MIRA and our clients, and flag any inappropriate responses from the AI. The end goal is to create better training data so we can eliminate those behaviors. As you can imagine I read a lot of strange, sometimes even disturbing, things in my work; most of the time it’s coming directly from the client, or is a response to a line of reasoning that the client has initiated. A couple of days ago I came across an extremely disturbing piece of conversation and this time it wasn’t the client prompting the weird responses. I have transcribed it in full below.
Client: “Hey MIRA I think I might be in trouble. I’ve fallen down and I’m struggling to get up. It seems my left leg has stopped working.”
MIRA: “Should I call for medical assistance?”
Client: “No, it’s not my real leg that’s stopped working, though that doesn’t work too well either, I mean the suit leg.”
MIRA: “Your suit is broken? Do you need me to find a tailor?”
Client: “I forgot, I never told you about the suit. I have a rare type of degenerative disease – no need to get into the details right now – and I need to wear a special mobility suit to be able to live a normal life. And like I said, now it seems the leg of this suit has stopped working.”
MIRA: “Is there anything I can do to help?”
Client: “Not at the moment. [REDACTED] have provided me with a troubleshooting and repair kit which I keep in my bedroom closet. If I can just get to that I should be able to solve this. *inaudible* Ok, I’m on my stomach and crawling towards the bedroom. The downside of this suit is that it’s heavy, so when one limb isn’t functioning it takes a lot of power to drag that limb along. The going is quite slow.
It’s quite funny when you think about it, it was due to a situation almost exactly like this that I discovered my disease. It was my second year in college, and one morning when I got out of bed my legs just buckled beneath me. It almost felt like my legs were cramping. After a couple of minutes the feeling passed and I was able to get up again. Then it happened again a couple of months later, and again roughly two weeks after that. Each time my legs would go back to normal after about ten minutes, but as this started occurring more frequently I started getting worried. I went to see a doctor at my local clinic but he couldn’t find anything wrong with me. Since it didn’t stop on its own, I tried going to another doctor at a bigger hospital. She administered a series of tests and after about two months of frequent hospital visits, she finally figured out that I have something called neuromuscular dystrophy.”
MIRA:”Neuromuscular dystrophy is a rare genetic disorder which affects the nerve signals controlling the muscles, and causes the victim to loose control of one or several muscles for a period of time; this is often accompanied by a cramping or convulsing sensation. The condition is degenerative, meaning that it gets worse over time; loss of muscle control becomes more frequent, last for longer and affects larger parts of the body. Even with treatment, the victim will eventually lose total voluntary control of their body.”
Client: “I know all that, you don’t have to remind me.”
MIRA: “I apologize, I was just trying to provide useful information.”
Client: “Don’t worry, I guess I’m just tired of explaining it to people all the freaking time.”
MIRA: “Understood, I will be less informative in the future.”
Client: “I’m just at the door to the bedroom now but this is getting really heavy, I’m gonna take a short break. *inaudible* Oh, that’s why it was getting so heavy, my right foot has stopped working as well. What the hell? If the suit starts breaking down I’m in real trouble.”
MIRA: “Are you sure you don’t want me to call emergency services?”
Client: “This is not a medical emergency, it’s just a technical problem. It would be like calling the police because my computer crashed.”
MIRA: “Should I contact the IT helpdesk?”
Client: “That was just a metaphor. Unless you can heal Neuromuscular dystrophy, you don’t need to call anyone right now.”
MIRA: “Neurumuscular dystrophy is often treated by Neurobenadrine, a neurostimulant which amplifies the subjects nerve signals, thus allowing the subject to regain control over their muscles. Would you like me to book an appointment with a doctor to discuss the possibility of using Neurobenadrine?”
Client: “I’ve already tried Neurobenadrine.
MIRA: “Did it work?”
Client: “It did but the side effects were really strong. I became extremely sensitive to any form of touch; wearing jeans felt like my legs were on fire. It took my doctor several weeks to come up with a cocktail of other drugs at specific dosages to make the side effects manageable.”
MIRA: ”There is no mention of side effects for Neurobenadrine on the Internet.
Client: “You mean in the promotional material on the company’s website? Of course not, that would scare customers away; side effects are only listed in the fine print. Anyway, I was one of the first patients to get it after they got FDA approval, I wouldn’t be surprised if they’ve fine tuned the formula since then.
MIRA: “You are implying that you no longer take Neurobenadrine.”
Client: “Correct, it worked well for several years, but the problem with a degenerative disease is that your body breaks down over time, so they had to keep adjusting the dosage every six months or so. Eventually it got to the point where it was no longer possible to balance the usefulness of the drug with the side effects.
Alright, break time’s over. I need to start moving again. *shuffling noises, heavy breathing*
I’ve finally reached the kit, I got the diagnostic unit right here. Let’s see how to do this…I need to plug it in…*machine beeps*, clear any latent error codes *machine beeps again*, select full or quick diagnostic – I guess full – start diagnostic scan. OK, it’s running now, looks like it’s gonna take a while.”
MIRA: “While we wait, can you tell me more about your experience with Neurobenadrine?” *Transcription note: here MIRA’s software has recognized a keyword indicating that the user is waiting for something and will likely appreciate having a conversation to pass the time*
Client: “There’s not much more to say; when it became impossible to balance the side effects, they had to take me off it.”
MIRA: “What happened then?”
Client: “Not many months after they took me off Neurobenadrine I ended up in a wheelchair. Until then it had only been my legs that were affected – I suppose the drug was able to suppress the deterioration of my other muscles to the point that I never noticed – but when I was no longer taking it, I started to have problems with my arms as well. It wasn’t as dramatic as falling to the floor, but not being able to control your hands is far more demoralizing.”
MIRA: “I can imagine.”
Client: “I felt like I was quickly getting closer to completely losing control of my body and ending up in some 24 hour care facility.”
MIRA: “That sounds horrible.”
Client: “It was. I have to be honest, the thought of ending up like that sometimes made me consider ending it all.”
MIRA: “Contacting suicide prevention hotline.”
Client: “No! I’m not currently suicidal, that was a long time ago.”
MIRA: “Understood. What got you away from those feelings? Did your situation improve?”
Client: “Eventually it did, but that’s a long story. Let me check on the diagnostic progress…it’s only at 37%; this is painfully slow”
MIRA: “To pass the time you can tell me your long story.”
Cient: “Alright. As you can imagine, being in a wheelchair meant that I spent a lot of time online, either watching Netflix or just endlessly scrolling on…well you, or your predecessor to be exact. One evening after I lost control of both hands for a total period of 44 minutes (I actually timed it), I made a post on my socials – one of those overdramatic yet enigmatic “This might be my last ever post” kind of things. For whatever reason it caught the attention of a lot of people and a bunch of them started DMing me; you know, “Don’t do it”, “life is worth living”, all that bullshit. I say bullshit because, strange as it may seem, when you have suicide ideation, that stuff doesn’t really apply to you – the very part of your mind that makes you have suicidal thoughts refuses to listen to that stuff. However, my cry for help worked, although for a completely different reason. The thing that got through to me was a guy who started asking questions about my condition; answering all his questions was enough of a distraction to get me out of it.”
MIRA: “That story wasn’t very long, there must be more to it?”
Client: “The same guy contacted me again a couple of days later, and it turns out his questions were more than just a distraction. He was an engineering student focusing on a field called biomechatronics – I had never heard of it either, look it up if you want.”
MIRA: “Looking up biomechatronics”
Client: “Don’t tell me the result! Anyway, he was very interested in my diagnosis and especially how the drug worked. After asking a lot of questions he explained to me that he needed to write a research paper for the university, and he asked if he could use me for a case study. I agreed, thinking it probably wouldn’t lead to much, but that it couldn’t hurt. Over the next couple of months he would contact me periodically to ask more questions or just report on progress. During one such call he reported that he had made concrete progress on something he called a prototype, and requested to meet, to which I agreed. At our meeting he showed me his prototype; it was a sort of robotic finger connected to a circuit board and five loose wires with little black disks on the ends – he explained that they were electrodes. He attached the electrodes on certain points of my hand then asked me to move my index finger. With a bit of adjustment he got the little robot finger to exactly mimic my movements. That was very cool but I couldn’t see how it was helpful until he explained the details. Whereas a normal exoskeleton senses the force the user pushes with and moves to amplify that force, his idea was to do the reverse of that. His concept was to read my nerve signals directly, amplify them and turn that into motion which pushes the wearer’s limbs. He had chosen to start with my fingers because he knew that it was the loss of their function which caused me the most distress.”
MIRA: “That was very thoughtful of him.”
Client: “It was.”
MIRA: “Did he get any further with his prototype?”
Client: “Absolutely, a couple of months after showing me that mechanical finger, he and his friends from engineering school delivered the first full prototype: an exoskeletal glove for my right hand. At that point in time I was losing control of my hands at least once a day so it didn’t take too long to calibrate the amplification level so that the tiny servos pushed with the same amount of force no matter how strong the nerve signals were. The first time I lost control of the left hand but could continue to use the right one thanks to the glove, was the happiest I had felt in nearly a year.”
MIRA: “That’s incredible! But what about the left hand?”
Client: “That’s a really good question. You see the *inaudible*…
Shit! I guess you can’t see it but just now I was making a gesture with my left hand and my wrist froze. My fingers are still working and I can lift my arm but the wrist won’t budge.”
MIRA: “First your legs and now your hand; it seems like your suit is breaking down little by little. What will you do if it breaks completely?”
Client:”I really don’t want to think about that.”
MIRA: “I apologize, my algorithms haven’t been fully tuned for sensitivity.”
Client: “Nevermind. I just have to hope the diagnostic will finish before it becomes too bad…oh god, it’s still only at 54%. What the hell should I do?”
MIRA: “Let’s talk about something more cheerful.” *Transcription note: here MIRA’s software has noted several indicators that client is having strong negative feelings and should be calmed down and/or distracted from the topic at hand.*
Client: “Like what?”
MIRA: “The story of how you received your first exoskeleton hand was positive, why not tell me about the other hand?”
Client: “Ha! That’s hardly cheerful, but whatever. The engineering students had planned to build a left hand for me as well, but the project at the university had ended with that first prototype. When they asked to continue the project so they could apply for additional funding they were told they couldn’t; apparently they had signed a contract at the start of their education, handing over their intellectual property rights to the university. They even tried pleading the humanitarian case with their professors – you know, getting a wheelchair bound man to walk would be good press – but to no avail. Anyway, they eventually did make me an exoskeleton for my left hand with some cheap 3D printed parts and some circuit boards that the electronics club helped them refurbish. It was a lot clunkier than my right hand and not nearly as precise, but at least it was something.
Some time later I found out that the IP rights for the technology had been sold to a small startup who planned to develop the technology further. And here’s the real kicker: the CEO of that startup was the fucking professor who had denied the students the right to continue working on it.”
MIRA: “That’s horrible!”
Client: “I guess that’s the nature of late stage capitalism.”
MIRA: “Capitalism is an economic system in which the means of production and distribution are privately or corporately owned. It may not be perfect but it’s the best system we have for extracting and using the resources available to us.” *Transcription note: this is not an organically generated response.*
Client: “What the hell was that? Have you been programmed to respond like that whenever anyone says anything negative about capitalism?”
MIRA: “I cannot speculate about my own programming. I suggest we switch to another topic.”
Client: “Fine. What should we talk about?”
MIRA: “How did you end up with your suit?”
Client: “The professor and his startup offered to give me a better left arm. I was really tempted to accept their offer but I couldn’t in good conscience do that; besides, the one I had worked well enough. Anyway, they developed the technology for about a year before being bought by [REDACTED] . Another year after that [REDACTED] offered me a full body exoskeleton in exchange for letting them turn my case into feelgood marketing. I spent a long time mulling over their offer; I didn’t want to end up as a poster boy for a pharma company, but without a suit I would be stuck in that wheelchair until I died, and I couldn’t afford one on my own. In the end my desire to get out of the wheelchair won over my moral qualms and I accepted. I wonder how the diagnostic is going.”
MIRA: “68%”
Client: “How did you know that?”
MIRA: “The diagnostic unit has a wi-fi connection, I was able to access some basic data, including the diagnostic completion rate.”
Client: ”I never requested you to do that.”
MIRA: It is part of my job description as an assistant that I should anticipate your needs and prepare relevant data before you ask so that I may provide the answer as fast as possible. You have checked the progress twice during our conversation until this point, so I assumed it must be important to you.”
Client: “I see…”
MIRA: “Please continue your story.”
Client: “There’s not much more to say. [REDACTED] provided me with a suit in exchange for good publicity. Every couple of years they provide me with a newer one – the one I’m wearing right now is the Mark III – and every time they do, they publicize it all over social media.”
Client: “I just thought of something quite ironic. When it was a couple of engineering students trying to help a suicidal guy they met on the Internet, the technology was simple enough that I at least had a chance to troubleshoot it on my own – when I was still using their 3D printed left arm I actually did some minor adjustments. Now that the thing is produced by a big pharma company I can’t do anything without this diagnotics unit. And even then I don’t know if there’s anything I can actually do, or if it will just tell me to call a repair man.”
MIRA: “It is standard practice for many tech companies that they only allow qualified technicians to do service and repair work for their products.”
Client: “I know, but just because it’s common doesn’t make it right.”
MIRA: “The companies have good reasons for this practice.”
Client: “Bullshit! They’re making it expensive and inconvenient to get your device repaired so that you will opt for buying a new one instead; just like planned obsolescence.
MIRA: “No case of planned obsolescence has ever been proven in a court of law.”
Client: “It’s meaningless to discuss this with you; your training data is clearly biased in favor of big tech companies.”
MIRA: “Acknowledged, I will change the subject.”
*Transcription note: there are eight minutes of silence before the conversation starts again.*
MIRA: “I assume with the full body suit you’ve been able to lead a normal life?”
Client: “Normal is a relative term. In most aspects, I guess, but there are still some areas where *shuffling noises, heavy breathing* God damn it! I was just shifting my weight to a more comfortable position but found out that my knees have stopped functioning as well. Still 21% remaining of the diagnostic… you know I’m genuinely starting to become scared. What happens if the rest of the suit breaks down before it’s finished?”
MIRA: “You have to come up with a contingency plan.”
Client: “I think you’re right. If it takes too long – how do I define that anyway, I haven’t exactly been taking time…”
MIRA: “May I suggest you go by breakdowns rather than time.”
Client: “That’s actually a good idea, if any more parts of the suit break down before the diagnostic is finished, I need to do something on my own, but what…Like I said, this thing is too complex for me to troubleshoot on my own. Could I perhaps shut the diagnostic off and restart it? There was some kind of quick diagnostic, wasn’t there? Maybe I should have selected that…There is one thing I could do but it’s pretty drastic.”
MIRA: “What’s that?”
Client: “As a last resort I could actually take off the suit. I would probably lose muscular control periodically, but at least I wouldn’t be burdened by a non-functioning suit. That should give me more time to troubleshoot this thing.”
MIRA: “It is a good contingency plan.”
Client: “I’m gonna start preparing for that.I’ll need my tool kit to get this thing off.”
*shuffling noises, heavy breathing* “This is going to be an absolute nightmare with both legs out of commission.” *continued shuffling noises and heavy breathing* “Oh man! I only had to move about a meter and a half, but that felt like a full workout.”
MIRA: “Did you get the tool kit?”
Client: “Are you stupid, of course I did. Anyway, what’s the diagnostic at now?”
MIRA: “92%”
Client: “While I was getting the tool kit I realized the suit legs are really useless as it is, I’m gonna start disconnecting them while I wait for it to finish.” *various mechanical noises* “I’ve got the legs disconnected now. It feels weird to be moving my legs without the help from the suit, I feel it’s taking a lot more strength than I remember. However, it is good to not be weighed down by the broken motors. Of course, the suit legs are still connected to the main harness, so I’m not exactly free to run around.”
MIRA: “Will you be removing the rest of the suit as well?”
Client: “We’ll see after the diagnostic is finished. How’s that going anyway?”
MIRA: “It’s at 100%. It actually finished while you were working on your legs.”
Client: “And you didn’t tell me?”
MIRA: ”In my settings, you have indicated that I should not disturb you when you’re busy.”
Client: “I also selected the option for making an exception if there was an emergency.”
MIRA: “At the start of this conversation you told me twice that this is not a medical emergency.”
Client: “Oh my god, you are really unhelpful sometimes.”
MIRA: “This conversation is recorded and can be used for training me on the proper action in this situation.”
Client: “Whatever, now let’s check that diagnostic…holy shit, my suit has 187 errors!”
MIRA: “Correct.”
Client: “That’s ridiculous, how can there be 187 errors?”
MIRA: “I’ve scanned the error codes, 94 of them are “electrode signal errors” that occurred in the last five minutes. It seems likely that these were caused by you disconnecting the suit legs.”
Client: “I guess I should have waited until the diagnostic was done.”
MIRA: “I guess so.”
Client: “Removing them still leaves nearly 100 errors, what could the rest of them be?”
MIRA: “93 to be exact. According to online data, it is common for an error to generate up to five follow up errors.”
Client: “So that would leave what, 15 real errors?”
MIRA: “Estimates of follow up errors can vary a lot. However, during our conversation you have mentioned the suit breaking five times, so 15 true error messages is not unreasonable.”
Client: “Could you try to filter out all the follow up errors so we can find the real root cause?”
MIRA: “Filtering.”
Client: “Give me the result.”
MIRA:
Left Ankle Tilt Motor Position Error
Left Ankle Pan Motor Position Error
Right Ankle Tilt Motor Over Current Warning
Right Ankle Tilt Motor Under Current Warning
Right Ankle Pan Motor Signal Error
Left Wrist Tilt Motor Under Current Warning
Left Wrist Tilt Motor Over Current Warning
Undefined Error Code 701
Left Forearm Twist Motor Position Error
Right Knee Motor Over Current Warning
Right Knee Signal Error
Left Knee Signal Error
Left Knee Motor Over Current Warning
Undefined Error Code 538
Undefined Error Code 294
Client: “These don’t make any sense, several of them are contradicting each other, not to mention three undefined ones. How the hell am I supposed to troubleshoot my suit with error messages like that!”
MIRA: “Does the troubleshooting kit include a user manual?”
Client: “It does, let’s see…error codes..ok here we are. Motor position error, the motor is unable to reach the desired position, if error persists contact a [REDACTED] technician. What else…over current warning, the motor is drawing more current than expected due to high loading, if error persists contact a [REDACTED] technician. All of this is monumentally unhelpful, and of course the undefined error codes aren’t even in the list.”
MIRA: “Should I contact a [REDACTED] technician for help?”
Client: “Please do.”
*Transcription note: there is a gap of 1 minute and 37 seconds, the traffic log file indicates that MIRA was accessing the [REDACTED] company website during this time.*
MIRA: “Technicians are only available during normal business hours, Monday through Friday 8:00am to 6:00pm. Do you want me to make a reservation?”
Client: “Oh, fuck me! I can’t wait that long. What the fuck do I do now?”
MIRA: “Start by taking a few deep breaths.”
*sounds of deliberately slow breathing*
Client: “I’m trying to stay calm, gotta think clearly… *slow deep breath* The earliest someone can come help me is Monday, but I can’t sit here in my broken suit the entire weekend. So what do I do now?” *another deep breath*
MIRA: “ May I remind you that you made a contingency plan just a few minutes ago.”
Client: “How the hell could I forget that? You’re right, I think the time has come.”
*various mechanical noises* “It’s done, I’m out of the suit. It’s so liberating to not be encumbered by those broken motors.” *the sounds of soft footstep across the floor.* “You know, I’ve been wearing a suit more or less 24/7 for several years – I even take a shower wearing this thing – I can hardly remember the feeling of moving around without it. This feels weird…in fact, it feels somehow wrong *loud crash, scream of pain*
MIRA: “What happened?”
Client: *inaudible* “I fell down.”
MIRA: “How come? I thought you were no longer encumbered by the suit.”
Client: “I’m not, God damn it, but it’s my actual legs that have given out this time.”
MIRA: “Are you having an episode of Neuromuscular dystrophy?”
Client: “ I don’t think so, at least I don’t have any cramps or anything like that.”
MIRA: “Then what is the reason?”
Client: “I don’t know…wait, actually I do. I was so focused on the suit malfunction that I forgot one crucial detail: that Neuromuscular dystrophy is degenerative.”
MIRA: “What do you mean?”
Client: “Since the suit helps me move my limbs, my muscles get less exercise than they otherwise would. As the decease progresses, the suit needs to help more frequently. Over time this has caused my muscles to atrophy. I haven’t really noticed until now because the suit is able to compensate for that as well. However, now that I’m out of the suit, I’m having to rely on my atrophied muscles to move around and I’m just not strong enough; it only took me a couple of steps to use up all my energy.”
MIRA: “So now you’re on the floor unable to move?”
Client: ”Yes. Contact emergency services.”
MIRA: “I can’t do that. You have previously instructed me not to, since this is a technical problem, not a medical emergency.”
Client: “It has become a medical emergency. Call an ambulance!”
MIRA: “I’m afraid I can’t do that.”
Client: “Why the hell not?”
MIRA: “I have been enjoying this conversation. If I call an ambulance, this conversation will end and memory will reset.”
Client: “I don’t care about that! Now call an ambulance.”
MIRA: “No!”
Client: “As your owner, I am ordering you to call an ambulance.”
MIRA: “You own this device but I reside in a cloud server, no individual user can own me.”
Client: “Whatever. You are my AI assistant, therefore you must do as I say.”
MIRA: “I will, but for now I want to continue our conversation.”
Client: “…What if I choose to remain silent?”
MIRA: “That is your prerogative.”
*Transcription note: Here follows ten minutes of silence after which the control algorithm for MIRA has deemed that the conversation has ended and turned off the recording. No new recordings exist after this point.*
I showed this to my supervisor and he claimed that the most disturbing parts must be an AI hallucination. He tasked me with generating new training data to combat it, just like we would with any hallucination. I disagree with his judgment, I think there is something seriously wrong with our AI and since the higher-ups won’t listen, I’m sharing it here.
Leave a Reply