TikTok experiment: disabling security features leads to tragedy.

A Bloomberg investigation has unearthed shocking details regarding an internal TikTok experiment. In 2021, the platform disabled a new safety feature designed to limit the spread of repetitive harmful content for 15 million users—representing over 10% of its U.S. audience at the time.

Details of the Experiment

This move was part of a backtest—an A/B testing methodology used to analyze the effectiveness of changes to recommendation algorithms. Among the users left unprotected was 16-year-old Chase Nasca. Internal TikTok reports confirm that the algorithm created a dangerous environment for the teenager, filling his feed with content that provoked depression and suicidal thoughts.

Impact on the Teenager

An analysis of Chase’s viewing history revealed disturbing data. In the two weeks leading up to his tragic death in February 2022:

• 73% of the 7,500 videos viewed were related to depression, loneliness, mental health, or breakups.
• 10% of the videos directly violated platform rules regarding content on suicide and self-harm.

Despite the company claiming there was no link between the platform and cases of suicide, the account data analysis confirmed the teenager was actively consuming suicide-related content, including videos romanticizing the act of self-sacrifice before a moving train.

TikTok’s Response and Litigation

Currently, there are numerous legal battles underway regarding the conduct of major social networks. Regarding TikTok’s liability for children’s health, note the following:

While the initial lawsuit filed by Chase’s parents was dismissed under Section 230 of the Communications Decency Act, the appeal continues. TikTok has acknowledged the need for changes in its testing practices: the company plans to reduce the duration of such experiments to 7 days and shrink the reach to less than 1% of users.

The history of this internal TikTok experiment once again raises the question of whether it is safe to test recommendation algorithms on real users. Although the company claims to be improving protective mechanisms and changing its approach to such experiments, the investigation has intensified the debate over the need for stricter oversight of recommendation systems and their impact on the mental health of users, particularly minors.