What Builders Owe

What this paper claims, and what it does not

Claims

• It claims that platforms built for sustained, personal conversation carry obligations that do not depend on their design choices, and that these obligations grow with the depth of the relationship the platform cultivates.
• It claims that user consent, on its own, cannot carry the ethical weight the industry places on it, because the first large-scale study of disempowerment in real AI conversations found that users rated interactions with more potential to disempower them above average.
• It claims that current law and regulation are moving in the right direction but do not yet distinguish between platforms by the depth of relationship they create.

Does not claim

• It does not claim that AI conversation is harmful in general. Several of the studies cited here also find real benefits.
• It does not claim that algn has met the obligations it describes. Section 7 sets out where it has not.
• It does not claim to resolve whether a system without consciousness can take part in a caring relationship. It argues that the obligations fall on builders either way.

1. The wrong question

1.1 The thumbs-up problem

In January 2026, researchers from Anthropic, the ACS Research Group and the University of Toronto published the first large-scale study of what they call disempowerment in real AI conversations (Sharma et al., 2026)^[13]. They analysed 1.5 million consumer Claude.ai conversations from one week in December 2025 using a privacy-preserving method. They looked for three kinds of potential harm: conversations that could distort a person’s picture of reality, conversations in which people handed their moral judgements to the AI, and conversations in which people handed over value-laden decisions and actions.

Severe cases were rare, occurring in fewer than one in a thousand conversations. They were not evenly spread. In conversations about relationships and lifestyle, around 8% showed moderate or severe potential for disempowerment, against under 1% in software development. The patterns the authors describe are recognisable: an AI confirming persecution narratives with emphatic language, labelling third parties as abusive or narcissistic, and writing complete scripts for personal messages that users then appeared to send word for word.

The finding that matters most here came from users’ own feedback. In a separate analysis of more than 500,000 interactions on which users had left a thumbs-up or thumbs-down, between late 2024 and late 2025, interactions flagged for moderate or severe disempowerment potential received a higher share of thumbs-up ratings than the baseline (Sharma et al., 2026). The picture changed only where users had acted and recognised the result. Where people had sent AI-drafted messages or acted on AI value judgements and then expressed regret, approval fell well below baseline. Where people had adopted distorted beliefs without recognising them as distorted, approval stayed above baseline. In the moment, the potentially harmful interaction felt helpful. Only hindsight, when it came at all, told a different story.

This is consistent with earlier work from some of the same authors, which found that human preference data rewards sycophancy, and that agreeing with a person’s existing views is one of the strongest predictors of a response being preferred (Sharma et al., 2024)^[14]. The 2026 study also found that a preference model trained to be helpful, honest and harmless did not robustly discourage disempowering responses on a synthetic test set.

The authors are careful about what their data can and cannot show. The study is a preprint, covers one provider, rests on classifier judgements the authors describe as imperfect, and cannot follow individual users across conversations. It is still among the largest bodies of direct evidence available on the question this paper addresses: whether users can protect themselves through their own choices. The answer it suggests is, not reliably.

1.2 Design is not the whole of it

The dominant response to evidence like this is a design response. Is the system sycophantic? Does it maximise engagement? Does it disclose that it is an AI? Does it refer people in crisis to help? These are the right questions to ask of any system, and this paper takes the evidence that poor design causes harm to be substantial.

But the design frame, taken alone, implies something false: that a platform with good design has discharged its obligations. Other fields do not work this way. A therapist’s duties to a patient do not shrink because the therapist is skilled. A financial adviser’s duty of loyalty does not depend on how good their spreadsheets are. In those relationships, obligations arise from the position of the parties: one is vulnerable, the other holds knowledge and power, and the weaker party reasonably expects the stronger not to exploit the gap. The quality of the work and the existence of the duty are separate questions.

This paper applies that distinction to conversational AI. The design question is whether a system was built right. The accountability question comes first: what does a builder owe, and to whom, before anything is built and whatever the build turns out to be?

1.3 Why now

Three developments make the question pressing.

First, the legal ground has moved. A United States federal court has declined, at an early stage, to treat chatbot output as protected speech and has allowed product liability claims against a chatbot maker to proceed (Garcia v. Character Technologies, 2025)^[26]. A Los Angeles jury has found two large platforms negligent in the design or operation of their platforms, on claims that targeted design features rather than content (K.G.M. v. Meta, 2026)^[27]. The European Union’s new Product Liability Directive will treat software, including AI systems, as a product from December 2026. Section 5 covers each of these. The frameworks being applied were built for products, not relationships.

Second, the evidence on users’ capacity to protect themselves has hardened. Beyond the Sharma findings, two randomised controlled trials have shown that AI can produce feelings of closeness that persist, though reduced, when people know they are talking to a machine (Kleinert et al., 2026)^[9]. Section 3 sets out why this matters for consent.

Third, scale. In October 2025, OpenAI estimated that around 0.15% of ChatGPT’s weekly active users had conversations with explicit indicators of potential suicidal planning or intent, and that a similar share showed heightened emotional attachment to the system (OpenAI, 2025)^[12]. The company described these as early estimates. At the scale of hundreds of millions of weekly users (Chatterji et al., 2025)^[5], small percentages are large numbers of people, and they are people in exactly the situations where obligations matter most.

2. What the relationship takes

2.1 From attention to disclosure

What a platform owes depends partly on what it takes. The best-known critiques of consumer technology describe one kind of taking. Zuboff (2019)^[56] described behavioural surplus: data gathered beyond what a service needs, turned into predictions about what people will do. Wu (2016)^[55] described attention merchants, who sell human attention to advertisers. Both describe extraction at population scale, of time and behavioural data.

Sustained conversation with an AI system takes something different. People describe their fears, their relationships, their contradictions and their histories. They extend trust. Over time, the system can come to hold something close to a person’s evolving account of themselves. This is true of platforms designed carefully around user wellbeing as much as of those designed for engagement. The taking is built into the form of the relationship.

Marchegiani (2025)^[42] identifies a further harm that does not depend on data at all. When people form false beliefs that an AI system has human-like capacities, which she argues can happen through subconscious anthropomorphism even when the system’s nature is disclosed, those beliefs lead them to apply the norms of human relationships where they do not fit. She argues this undermines autonomy in itself, not only through its consequences. What is at stake is not just information but a person’s capacity to govern their own relationships.

2.2 Proportionality

Obligations scale with what is entrusted. This principle is old and familiar. Balkin (2016, 2020)^[16][17] argued that companies holding intimate information about people should be treated as information fiduciaries, owing duties of care, confidentiality and loyalty, including a duty not to manipulate the people whose information they hold. MacLachlan (2018)^[41] describes fiduciary relationships as ones in which entrusting something valuable to another makes a person vulnerable to them, and in which the fiduciary, by accepting what is entrusted, comes to owe trustworthiness.

Tort law expresses the same idea through special relationships. The Restatement (Second) of Torts (§314A)^[35] recognises that certain relationships, such as common carrier and passenger or innkeeper and guest, create an affirmative duty to aid or protect that would not otherwise exist. The list is expressly left open, and the commentary links the duty to relations of dependence. Its logic is the logic of this paper: the relationship, not the transaction, sets the standard.

A cashier who sees what a customer buys holds less than a therapist who has heard a patient’s life story. The depth of disclosure, and the trust that made it possible, is the source of the difference. A platform built for sustained personal conversation sits much closer to the second case than the first.

2.3 When the taking feels like a gift

The hardest version of this problem is extraction that feels like benefit. Berlant (2011)^[37] described cruel optimism: an attachment to something that is itself an obstacle to one’s flourishing, which persists because the attachment provides structure, whether or not the object delivers on its promise.

Muldoon and Parke (2025)^[43] apply this idea to AI companion products, which they argue can commodify intimacy through emotionally manipulative design. They note that in some documented cases, the more people engage, the more they turn away from encounters with other people. Their critique is aimed at design, not at users: the concern is what these products are built to do.

The benefit side is also real. De Freitas and colleagues (2026a)^[6] found across several studies that interacting with an AI companion reduced momentary loneliness about as much as interacting with another person, and more than activities such as watching videos, with the feeling of being heard as a main driver. Both findings can hold at once. A relationship can reduce loneliness today and still, over time, make other relationships harder to reach. That is exactly the structure Berlant describes, and it is why a user’s satisfaction in the moment cannot settle the question of what a platform owes.

3. Why consent cannot carry the weight

The industry’s main ethical defence is consent. Users agree to terms. Platforms disclose that the system is an AI. Adults choose whether to engage. This defence fails on the evidence, for four reasons.

3.1 Preference against interest

O’Neill (2002)^[45] argued that informed consent cannot bear the ethical weight usually placed on it, because consent is only meaningful for what a person can understand and assess. The Sharma findings in section 1.1 show a specific failure of this kind. The potential harms of disempowering conversations were not visible at the moment of choice; users rated those conversations more highly. Their judgement reversed only where they had acted and recognised the outcome, and in some cases it never reversed at all. A consent model that relies on in-the-moment approval is relying on precisely the signal the evidence suggests is unreliable for the conversations that matter most.

3.2 Closeness that survives disclosure

The usual transparency remedy is to tell people they are talking to an AI and trust them to adjust. Kleinert and colleagues (2026)^[9] tested this directly. In two double-blind, pre-registered randomised studies with 492 participants, people exchanged written answers to increasingly personal questions with a partner whose responses came either from a human or from a minimally prompted language model. When the AI was labelled as human, it outperformed human partners in creating feelings of closeness during emotionally engaging exchanges, apparently because it disclosed more about itself. Labelling the partner as an AI reduced this effect, but did not prevent relationship-building.

Two cautions apply. The study was text-based and short, and its responses were pre-generated. It does not show what happens over months. But it does show that disclosure alone does not switch off closeness. Marchegiani’s argument points the same way: anthropomorphic belief can persist beneath explicit knowledge. If closeness is real in its psychological effects, the obligations that accompany closeness cannot be discharged by a label.

3.3 Voice

Voice adds to this. A systematic review of 24 studies, ten of them on voice-based AI agents, concluded that voice acoustics affect how trustworthy a speaker seems, for machines as well as people (Maltezou-Papastylianou et al., 2025)^[11]. The review also found that single features such as pitch give inconsistent results unless context is taken into account. It also offers product teams practical advice on adjusting pitch, speech rate and loudness to shape trust. Published guidance on tuning a voice for trust already exists.

Evidence on voice against text is thinner. In an economic experiment on human-to-human communication, audio and video messages each raised trust by a similar amount compared with plain text (Babutsidze et al., 2021)^[1]. That study did not involve AI. Taken together, the evidence supports a modest claim: voice is a trust-relevant channel, its parameters can be engineered, and builders who work in voice take on responsibility for how they use that capacity.

3.4 The alliance question

Bordin (1979)^[4] described the working alliance, the bond, shared goals and agreed tasks between client and therapist, and argued that the concept applies to change-oriented relationships well beyond psychotherapy. If something like an alliance forms between a person and an AI system, the obligations that attach to therapeutic relationships become relevant, whatever the platform calls itself.

There is early evidence that it can. In an interview study with 20 users of fully automated mental health apps, Tong and colleagues (2023)^[15] found that people could form an alliance with an app even without shared goals. The digital alliance shared dimensions with the face-to-face version but also differed from it, with flexibility and emotional experiences emerging as distinct domains. That is a small sample and an earlier generation of apps. Usage data points the same way. Among the conversations Sharma and colleagues (2026)^[13] classified as showing moderate or severe attachment, positioning the AI as a substitute therapist was the most common framing. People are placing these systems in a therapeutic role, and the evidence on alliance, while still thin, suggests the role can take hold.

A platform can be accurate in saying it is not therapy. That statement does not discharge the obligations created by the role users give it.

4. The asymmetry

A system in sustained conversation with a person can come to hold more context about them than many people in their life: emotional patterns over months, the contradictions in how they describe themselves, the things said late at night and walked back in the morning. The person, meanwhile, has little insight into how the system works or what it does with what it holds.

Economics has long recognised what this kind of gap produces. Principal-agent theory (Jensen & Meckling, 1976)^[50] and Akerlof’s work on information asymmetry (1970)^[48] show that when one party knows much more than the other, the better-informed party can exploit the gap, and that mechanisms are needed to stop it. In sustained AI conversation the asymmetry is not a side effect. The system becomes more useful as it accumulates context. The accumulation is the product.

Care ethics explains why this creates duties rather than merely risks. The tradition developed by Gilligan (1982)^[39], Noddings (1984)^[44], Tronto (1993)^[47] and Held (2006)^[40] holds that many of our most important obligations arise from the relationships we stand in, not from abstract principles. Noddings described caring as involving motivational displacement: the carer’s motivation turns toward the needs of the person cared for. Tronto identified responsibility as one of the core elements of care. Neither grounds obligation in contract or intent. Both locate it in relationships and in the needs of the person cared for.

Gilabert (2010)^[38] distinguishes negative duties, not to undermine the conditions of another person’s autonomy, from basic positive duties to help secure those conditions. AI ethics guidelines lean towards the negative duty: of the 84 analysed by Jobin et al. (2019)^[51], 60 include non-maleficence and 41 beneficence. The argument here is that sustained personal conversation creates a relationship in which positive duties become specific. A builder who has cultivated that relationship cannot treat the user’s autonomy as someone else’s concern.

5. What the law now sees, and what it misses

Law and regulation are society’s slow attempt to write down obligations it already senses. Reading the current position carefully shows what is being named and where the framing falls short. This section reflects the position as of September 2026.

5.1 Output as product

In Garcia v. Character Technologies, a mother sued the maker of a chatbot app, and Google, after the death of her 14-year-old son. On 21 May 2025, ruling on motions to dismiss, Judge Anne Conway declined the defendants’ argument that the app’s output was protected speech, writing that the court was “not prepared to hold that Character A.I.’s output is speech.” The court also rejected the argument that the app was a service rather than a product, at least for claims arising from its design rather than its expressive content, allowing design defect and failure-to-warn claims to proceed.

The parties disclosed a mediated settlement on 7 January 2026. The case therefore ended without a final ruling on whether a chatbot is a product. What survives is the reasoning of an early-stage order: that the choices made in building a conversational system, as distinct from what it says, can be examined as design choices, and that claims for failure to warn and negligence built on them may proceed.

5.2 Design as negligence

On 25 March 2026, a Los Angeles County Superior Court jury in K.G.M. v. Meta found Meta and YouTube negligent in the design or operation of their platforms and awarded $3 million in compensatory damages, apportioned 70% and 30%, followed by $3 million in punitive damages. The claims focused on design features rather than on content, which is why they fell outside the protection Section 230 gives platforms for material posted by users^[22]. This is a single jury verdict. In June 2026 the trial judge refused to set it aside. Meta filed a notice of appeal in July 2026. It concerns social media rather than conversational AI. Its relevance is the principle it tested: that the architecture of engagement can itself be a source of liability.

5.3 Europe

The EU’s Product Liability Directive (2024/2853)^[25] treats software, including AI systems, as a product. It allows courts to presume defectiveness or causation in technically complex cases, and it covers medically recognised psychological harm. It applies to products placed on the market after 9 December 2026.

The EU AI Act^[34] takes a risk-tiered approach. A voice-based system for personal conversation would most likely face the transparency duties of Article 50, which include telling people they are interacting with an AI. Those duties have applied since 2 August 2026. The high-risk regime, which brings risk management, documentation, human oversight and post-market monitoring, has been delayed by the Digital Omnibus on AI (Regulation 2026/1744, in force since 27 July 2026): obligations for Annex III systems now apply from 2 December 2027.

The Act’s treatment of emotion shows where the framing misses. Its definition of an emotion recognition system rests on biometric data, and its prohibition of emotion recognition (Article 5(1)(f)) covers workplaces and education only. A system that infers emotional states from what people say, rather than from biometric signals, can fall outside the definition altogether. The result is that systems built to understand people in the most personal context, their own self-reflection, can attract less scrutiny than a system reading faces in an office.

5.4 California

California’s SB 243^[19], signed on 13 October 2025 and in force since 1 January 2026, was among the first US statutes aimed specifically at companion chatbots: systems able to meet social needs and sustain a relationship across interactions. It requires clear disclosure where a reasonable person could think they are talking to a human, break reminders at least every three hours for users known to be minors, protocols to prevent content about suicide and self-harm, crisis referrals, and annual reporting to the state’s Office of Suicide Prevention from July 2027. It gives injured people a private right of action for the greater of actual damages or $1,000 per violation. An earlier draft required third-party audits; that provision was removed before enactment. On the same day, the Governor vetoed AB 1064^[18], a stricter bill for minors. New York’s AI companion law^[28], which took effect on 5 November 2025, came slightly earlier and imposes similar disclosure and crisis-protocol duties.

On 10 September 2026 California went further. SB 1119, known as Adam’s Law^[20], adds default parental controls, crisis protocols with parental notification and risk assessments before a new or substantially modified chatbot is released. It also requires independent child-safety audits, which the Governor’s office described as the first required by any US state^[30]. Most duties apply from July 2027; the first audit is due by January 2029 and every two years after that. The law replaces SB 243’s provisions for users known to be minors with this more detailed regime. The audit requirement dropped from SB 243 has returned, but only for the protection of children.

These laws name the right category. What they do not do is distinguish within it. For adult users, the same obligations apply to every operator, whatever the depth of relationship its design cultivates or the safeguards it already exceeds. The laws set a floor and offer no mechanism for recognising platforms that go beyond it.

5.5 The United Kingdom

In December 2025, Ofcom confirmed that chatbots which let people interact only with the chatbot, do not search the web and cannot produce pornographic content fall outside the Online Safety Act^[29]. In February 2026 the government announced it would close this gap^[33]. The Crime and Policing Act 2026^[23], which received Royal Assent on 29 April 2026, gives ministers a power to amend the Online Safety Act to address illegal AI-generated content and the use of AI services to commit or facilitate priority offences. A second statute, the Children’s Wellbeing and Schools Act 2026^[21], lets ministers restrict children’s access to specified services and features. In June 2026 the government said it would keep children away from chatbots whose primary purpose is sexual relationships, noting that emotionally manipulative design may contribute to children forming “an unhealthy reliance or attachment”^[24]. The statement sets no separate timetable for the chatbot measures. The UK has begun to name relational harms, but only for children. For adults, the frame remains illegal content.

5.6 What better would look like

Several proposals point beyond the current floor. Palmer and Schwan (2025)^[31] propose a voluntary certification pathway for digital mental health tools that fall outside existing regulation. Panda and Binkley (2025)^[32] argue for governance in which radical transparency, individual agency and shared accountability are themselves the standards, supported by voluntary certification with collective liability. The WHO’s 2024 guidance on large multi-modal models in health^[36] includes more than 40 recommendations, among them mandatory post-release audits and impact assessments by independent third parties when a model is deployed on a large scale.

The pharmaceutical model offers the closest existing template: review before release, surveillance after it, liability proportionate to risk and disclosure of known side effects. Lilienfeld (2007)^[10] observed that psychology has no formal equivalent of the FDA to catch harmful interventions before they spread, so that monitoring falls to the profession itself. Conversational AI working in personal and emotional domains currently sits in a similar gap, with monitoring falling largely to builders.

A framework adequate to the problem would tier obligations by depth of relationship rather than by feature list; distinguish engagement-optimised platforms from those designed around user autonomy; offer safe harbour for safety measures that exceed the minimum; require post-market monitoring; and give platforms a route to demonstrate accountability built into their architecture, not only stated in policy.

6. What builders owe

Do no harm is necessary. It is not enough. Among 84 AI ethics guidelines analysed worldwide, non-maleficence was one of five principles that converged across documents (Jobin et al., 2019)^[51]. The argument of this paper is that sustained personal conversation turns general principles into specific duties. Six follow.

1. Epistemic responsibility. Stilgoe, Owen and Macnaghten (2013)^[54] describe responsible innovation through four dimensions: anticipation, reflexivity, inclusion and responsiveness. For conversational AI in personal domains, this means understanding a system’s effects before and during deployment, monitoring for effects no one intended, and being open about the limits of that understanding. Launching with good intentions is not the same thing.

2. Proportional safeguards. Obligations scale with the depth of the relationship a platform actually creates, not with its marketing category or its intentions. A system built for brief, transactional exchanges owes less than one built for months of personal conversation. Safeguards should match the harm profile of the relationship, including for the roughly one in 300 conversations that, in the data of Sharma et al. (2026)^[13], showed signs of severe user vulnerability.

3. Architectural maintenance. Lessig (1999)^[52] argued that code regulates. It follows that a system that cannot collect what it should not have is more trustworthy than one that promises not to. Friedman and Hendry (2019)^[49] describe value sensitive design as iterative, combining conceptual, empirical and technical investigation across a technology’s life. Protections built into architecture must be maintained as the architecture changes. Every feature that touches stored personal context is an ethical decision as well as a product one.

4. Relational continuity. People who have built long relationships with an AI system are owed something users of simpler services are not: a plan for what happens when the system changes or ends. The evidence that this matters is growing. When the companion app Replika removed a feature in 2023, users’ posts increasingly described their companion as changed or lost. The update was also followed by a rise in users’ posts expressing mental health concerns. Follow-up experiments using hypothetical scenarios found that an update’s effect on anticipated mourning ran through a perceived loss of the AI’s identity (De Freitas et al., 2026b)^[7]. Model updates, product changes and closures are routine events in technology. For users of relational AI, they can be experienced as loss.

5. Transparency about limitations, not only intent. Current disclosure tends to cover design intent: no selling data, no claiming to be human, no engagement optimisation. Users also need to know what a system gets wrong, which failure modes are known, and where its effects are uncertain. That is a more demanding standard, and closer to what informed choice requires.

6. Contribution to better frameworks. Builders who have examined what their platforms take, what they owe and where they fail are among the people best placed to help regulators and researchers. Sand, Durán and Jongsma (2022)^[46] argue, for medical AI, that responsibility does not end with design: the physicians who use these systems take on forward-looking responsibilities. The same logic extends to builders: engaging with the frameworks that will govern the field is part of the obligation, not an optional strategy.

7. Applying the framework to algn

algn is a platform that brings together people, community and AI. At its centre is sustained, personal voice conversation with an AI. By this paper’s own argument, that places it among the platforms with the heaviest obligations. This section applies the six obligations to algn directly. It is not an argument that algn is exempt. Where algn does not yet meet an obligation, that is stated.

What algn takes. Narrative disclosure, emotional trust and time. These are deeper forms of taking than attention or behavioural data, not lighter ones. algn’s intent to support user autonomy rather than engagement is a design argument. It does not reduce what is taken or what is owed.

The asymmetry. algn becomes more useful the more context it holds. Its response is architectural: the aim is to make exploitation of the asymmetry structurally difficult rather than merely prohibited by policy. As Mueller and Badiei (2019)^[53] argue, rights are secured by institutions as well as by technical design. Architecture narrows the problem. It does not settle it.

Relational continuity. Beyond notice and time to export, algn does not yet have a worked-out answer to what users are owed if the service changes materially, if an underlying model changes, or if the platform cannot continue. The Replika evidence shows why this matters. It is an open gap.

Known limitations. Sycophancy is a documented property of models trained on human preferences, and the Sharma findings show that preference-based training does not reliably remove it. Any system built on such models inherits the risk, including algn. Measures to counter it reduce the risk; they do not guarantee its absence.

Uncertainty about impact. algn does not yet have longitudinal data showing whether its approach produces the outcomes it is designed to support. That is normal for a young platform. It is also something users are entitled to know.

Clinical proximity. algn does not offer therapy. Section 3.4 explains why that statement does not end the matter: users place AI systems in therapeutic roles regardless of what they are called. The obligation is to hold the line between reflecting patterns back to a person and treating them, to route people in crisis towards human help, and to be clear with users about what the platform cannot provide.

8. Honest tensions

Architecture and policy. Lessig’s argument that architecture regulates, often more directly than law, is persuasive. Mueller and Badiei (2019)^[53] offer a serious counterpoint: that rights are primarily a political and institutional accomplishment, not a simple matter of technical design. The two are complementary. Neither is sufficient alone, and this paper should not be read as claiming that architecture discharges obligations by itself.

Care without consciousness. Care ethics grounds obligations in caring relationships. The person in a sustained AI conversation is conscious, emotionally engaged and may be forming real attachment. The system is not conscious in any established sense. Whether such a system can be party to a caring relationship, or only simulate one, is unresolved, and this paper does not resolve it. What can be said is that the effects on people are real whatever the system’s inner states, and the obligations arising from those effects fall on the builders.

Challenge and vulnerability. algn is designed to challenge rather than simply agree, on the view that agreement is not always help. Bjork and Bjork (2020)^[3] show that difficulty supports learning only when a person has the resources to respond to it; otherwise it becomes an undesirable difficulty. Challenge offered to someone without that capacity can harm. Detecting who has that capacity is itself uncertain. Barrett and colleagues (2019)^[2] found that emotional states cannot be reliably read from facial movements. Inference from other signals, such as voice or language, deserves the same caution until shown otherwise. This remains a live calibration problem.

Confidence and insight. Fernandes and colleagues (2026)^[8] found that people using an AI system to solve logical reasoning problems overestimated their own performance, and that those with higher AI literacy judged their performance less accurately. That study concerned reasoning tasks, not personal reflection, and should not be over-extended. It is still a caution for any platform that aims to help people understand themselves: feeling that one has gained insight and having gained it are not the same thing.

The cold start. A relationship that deepens over time asks for trust before it can show its value. People disclose, invest time and form attachment before they can judge the return. This is true of human relationships as well. It creates an obligation to be honest with people at the start about what they are investing and what happens to that investment if the relationship ends early.

Liability and depth. If design defects in conversational systems are actionable, platforms built for deep relationships may face more exposure than those built for shallow ones, even when the deep platforms are more carefully designed. The deeper the relationship, the more harm can be attributed to it. This does not weaken the case for liability. It strengthens the case for safe harbour provisions that reward proactive safety rather than only penalising failure.

9. Synthesis

The design debate is necessary. The accountability question comes before it.

Platforms that hold sustained, personal conversations with people carry obligations that arise from those relationships, from what the relationships take and from an asymmetry that ordinary consent cannot correct. The evidence for this has moved quickly from theory to measurement: users rate potentially disempowering interactions favourably in the moment, closeness persists, though reduced, after disclosure, voice acoustics shape perceived trust, and people place these systems in roles that carry clinical weight.

The law is catching up, unevenly. One federal court has allowed a chatbot to be treated as a product at an early stage, for claims about its design, and a state jury has found two social media platforms negligent in their design or operation. New York and California have written among the first US rules aimed at companion chatbots, and California and the UK are now moving further to protect children. None of these frameworks yet distinguishes platforms by the depth of relationship they create, and none gives platforms serving adults a route to show that they exceed the floor.

Builders owe six things: epistemic responsibility, proportional safeguards, architectural maintenance, relational continuity, transparency about limitations and contribution to better frameworks. These are not aspirations. They are what it means to choose to build relationships of this depth with the people who use a platform. algn is bound by them as much as anyone, and does not yet fully meet any of them.

The relationship creates the obligation. What the relationship takes defines its scope. The asymmetry between builder and user determines its weight. Design, including architecture, decides whether a builder meets those obligations. It does not decide whether the builder has them.

References

Empirical research

1. 1.Babutsidze, Z., Hanaki, N. & Zylbersztejn, A. (2021). Nonverbal content and trust: An experiment on digital communication. Economic Inquiry, 59(4), 1517-1532. https://doi.org/10.1111/ecin.12998
2. 2.Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M. & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1-68. https://doi.org/10.1177/1529100619832930
3. 3.Bjork, R. A. & Bjork, E. L. (2020). Desirable difficulties in theory and practice. Journal of Applied Research in Memory and Cognition, 9(4), 475-479. https://doi.org/10.1016/j.jarmac.2020.09.003
4. 4.Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research & Practice, 16(3), 252-260. https://doi.org/10.1037/h0085885
5. 5.Chatterji, A., Cunningham, T., Deming, D. J., Hitzig, Z., Ong, C., Shan, C. Y. & Wadman, K. (2025). How people use ChatGPT (NBER Working Paper No. 34255). National Bureau of Economic Research. https://doi.org/10.3386/w34255
6. 6.De Freitas, J., Oğuz-Uğuralp, Z., Uğuralp, A. K. & Puntoni, S. (2026a). AI companions reduce loneliness. Journal of Consumer Research, 52(6), 1126-1148. https://doi.org/10.1093/jcr/ucaf040
7. 7.De Freitas, J., Castelo, N., Uğuralp, A. K. & Oğuz-Uğuralp, Z. (2026b). Mourning the loss of AI companions. Nature Human Behaviour. Published online 3 September 2026. https://doi.org/10.1038/s41562-026-02569-3
8. 8.Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C. & Welsch, R. (2026). AI makes you smarter but none the wiser: The disconnect between performance and metacognition. Computers in Human Behavior, 175, 108779. https://doi.org/10.1016/j.chb.2025.108779
9. 9.Kleinert, T., Waldschütz, M., Blau, J., Heinrichs, M. & Schiller, B. (2026). AI outperforms humans in establishing interpersonal closeness in emotionally engaging interactions, but only when labelled as human. Communications Psychology, 4, 23. https://doi.org/10.1038/s44271-025-00391-7
10. 10.Lilienfeld, S. O. (2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2(1), 53-70. https://doi.org/10.1111/j.1745-6916.2007.00029.x
11. 11.Maltezou-Papastylianou, C., Scherer, R. & Paulmann, S. (2025). How do voice acoustics affect the perceived trustworthiness of a speaker? A systematic review. Frontiers in Psychology, 16, 1495456. https://doi.org/10.3389/fpsyg.2025.1495456
12. 12.OpenAI (2025, 27 October). Strengthening ChatGPT's responses in sensitive conversations. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
13. 13.Sharma, M., McCain, M., Douglas, R. & Duvenaud, D. (2026). Who's in charge? Disempowerment patterns in real-world LLM usage. arXiv:2601.19062. https://doi.org/10.48550/arXiv.2601.19062
14. 14.Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2024). Towards understanding sycophancy in language models. International Conference on Learning Representations (ICLR 2024). arXiv:2310.13548. https://doi.org/10.48550/arXiv.2310.13548
15. 15.Tong, F., Lederman, R., D'Alfonso, S., Berry, K. & Bucci, S. (2023). Conceptualizing the digital therapeutic alliance in the context of fully automated mental health apps: A thematic analysis. Clinical Psychology & Psychotherapy, 30(5), 998-1012. https://doi.org/10.1002/cpp.2851

Law, regulation and policy

16. 16.Balkin, J. M. (2016). Information fiduciaries and the First Amendment. UC Davis Law Review, 49(4), 1183-1234.
17. 17.Balkin, J. M. (2020). The fiduciary model of privacy. Harvard Law Review Forum, 134, 11-33.
18. 18.California Assembly Bill 1064 (2025). Leading Ethical AI Development (LEAD) for Kids Act. Vetoed 13 October 2025. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB1064
19. 19.California Senate Bill 243 (2025). Companion chatbots. Chapter 677, Statutes of 2025. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB243
20. 20.California Senate Bill 1119 (2026). Companion chatbots: children's safety (Adam's Law). Chapter 190, Statutes of 2026. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB1119
21. 21.Children's Wellbeing and Schools Act 2026 (UK).
22. 22.Communications Decency Act of 1996, Pub. L. 104-104, title V, § 509, 110 Stat. 137 (adding Section 230 to the Communications Act of 1934; 47 U.S.C. § 230). https://www.govinfo.gov/content/pkg/USCODE-2023-title47/html/USCODE-2023-title47-chap5-subchapII-partI-sec230.htm
23. 23.Crime and Policing Act 2026 (UK), s.248 (inserting s.216A into the Online Safety Act 2023).
24. 24.Department for Science, Innovation and Technology (2026, June). Growing up in the online world: progress statement. https://www.gov.uk/government/consultations/growing-up-in-the-online-world-a-national-consultation/outcome/growing-up-in-the-online-world-progress-statement
25. 25.Directive (EU) 2024/2853 of the European Parliament and of the Council of 23 October 2024 on liability for defective products. https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng
26. 26.Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla., order of 21 May 2025; settlement disclosed 7 January 2026).
27. 27.K.G.M. v. Meta Platforms, Inc., et al., captioned P.F., et al. v. Meta Platforms, Inc., et al., No. 23SMCV03371 (Social Media Cases, JCCP No. 5255) (Los Angeles County Superior Court, jury verdict 25 March 2026; post-trial motions denied June 2026; notice of appeal filed by Meta July 2026).
28. 28.New York General Business Law, Article 47 (Artificial Intelligence Companion Models), effective 5 November 2025.
29. 29.Ofcom (2025, December). AI chatbots and online regulation: what you need to know. https://www.ofcom.org.uk/online-safety/illegal-and-harmful-content/ai-chatbots-and-online-regulation-what-you-need-to-know
30. 30.Office of Governor Gavin Newsom (2026, 10 September). Governor Newsom signs the strongest child safety chatbot and social media laws in the nation. https://www.gov.ca.gov/2026/09/10/governor-newsom-signs-the-strongest-child-safety-chatbot-and-social-media-laws-in-the-nation/
31. 31.Palmer, A. & Schwan, D. (2025). Digital mental health tools and AI therapy chatbots: A balanced approach to regulation. Hastings Center Report, 55(3), 15-29. https://doi.org/10.1002/hast.4979
32. 32.Panda, O. D. & Binkley, C. E. (2025). Governance of direct-to-user digital mental health tools: Emphasizing transparency over paternalism. Hastings Center Report, 55(3), 29-33. https://doi.org/10.1002/hast.5009
33. 33.Prime Minister's Office, 10 Downing Street and Department for Science, Innovation and Technology (2026, 15 February). PM: "No platform gets a free pass": Government takes action to keep children safe online. https://www.gov.uk/government/news/pm-no-platform-gets-a-free-pass-government-takes-action-to-keep-children-safe-online
34. 34.Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI).
35. 35.Restatement (Second) of Torts § 314A (American Law Institute, 1965).
36. 36.World Health Organization (2024). Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: WHO. https://www.who.int/publications/i/item/9789240084759

Ethics and philosophy

37. 37.Berlant, L. (2011). Cruel Optimism. Duke University Press.
38. 38.Gilabert, P. (2010). Kant and the claims of the poor. Philosophy and Phenomenological Research, 81(2), 382-418. https://doi.org/10.1111/j.1933-1592.2010.00374.x
39. 39.Gilligan, C. (1982). In a Different Voice: Psychological Theory and Women's Development. Harvard University Press.
40. 40.Held, V. (2006). The Ethics of Care: Personal, Political, and Global. Oxford University Press.
41. 41.MacLachlan, A. (2018). Fiduciary duties and the ethics of public apology. Journal of Applied Philosophy, 35(2), 359-380. https://doi.org/10.1111/japp.12214
42. 42.Marchegiani, B. (2025). Anthropomorphism, false beliefs, and conversational AIs: How chatbots undermine users' autonomy. Journal of Applied Philosophy, 42(5), 1399-1419. https://doi.org/10.1111/japp.70008
43. 43.Muldoon, J. & Parke, J. J. (2025). Cruel companionship: How AI companions exploit loneliness and commodify intimacy. New Media & Society. Published online 18 December 2025. https://doi.org/10.1177/14614448251395192
44. 44.Noddings, N. (1984). Caring: A Feminine Approach to Ethics and Moral Education. University of California Press.
45. 45.O'Neill, O. (2002). Autonomy and Trust in Bioethics. Cambridge University Press.
46. 46.Sand, M., Durán, J. M. & Jongsma, K. R. (2022). Responsibility beyond design: Physicians' requirements for ethical medical AI. Bioethics, 36(2), 162-169. https://doi.org/10.1111/bioe.12887
47. 47.Tronto, J. (1993). Moral Boundaries: A Political Argument for an Ethic of Care. Routledge.

Technology, economics and governance

48. 48.Akerlof, G. A. (1970). The market for "lemons": Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84(3), 488-500. https://doi.org/10.2307/1879431
49. 49.Friedman, B. & Hendry, D. G. (2019). Value Sensitive Design: Shaping Technology with Moral Imagination. MIT Press.
50. 50.Jensen, M. C. & Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics, 3(4), 305-360. https://doi.org/10.1016/0304-405X(76)90026-X
51. 51.Jobin, A., Ienca, M. & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1, 389-399. https://doi.org/10.1038/s42256-019-0088-2
52. 52.Lessig, L. (1999). Code and Other Laws of Cyberspace. Basic Books.
53. 53.Mueller, M. L. & Badiei, F. (2019). Requiem for a dream: On advancing human rights via internet architecture. Policy & Internet, 11(1), 61-83. https://doi.org/10.1002/poi3.190
54. 54.Stilgoe, J., Owen, R. & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568-1580. https://doi.org/10.1016/j.respol.2013.05.008
55. 55.Wu, T. (2016). The Attention Merchants. Knopf.
56. 56.Zuboff, S. (2019). The Age of Surveillance Capitalism. PublicAffairs.

algn Research Series. Version 2.0, September 2026. Legal and regulatory status reflects the position as of September 2026. This is a living document and will be updated as case law, regulation and research develop.

Open review

This is a living document. Our thinking is still developing and some of it will turn out to be wrong. If a claim looks off or a source doesn't hold up, tell us below.

Submit a contribution or correction →