How YouTube develops its policies to curb harmful content & garner business success

Since the earliest days of YouTube, there have been Community Guidelines to establish what’s allowed on the platform. These rules of the road gave rise to creative expression, while prioritising the protection of the entire YouTube community from harmful content. In a blog, Matt Halprin, Global Head of Trust & Safety, YouTube (Google), and Jennifer Flannery O'Connor, Vice President, Product Management, YouTube, write that this balance is critical for allowing new voices to participate and to promote the sharing of ideas. It’s also necessary to ensure YouTube’s long-term success as a business, as the platform’s advertising partners fundamentally do not want to be associated with harmful content.

Halprin and O'Connor further write that a few questions have regularly cropped up around how YouTube decides where to draw these lines or why it can take so long to develop and launch new policies (for a broader overview of how we think about our responsibility efforts, see here). In this blog, Halprin and O'Connor shed more light on how YouTube develops its policies and the processes that go into enforcing them.

Determining what policy updates are needed

The world moves quickly and the policies need to keep up. That’s why YouTube regularly reviews its policies to make sure that — similar to the laws that govern civil society — they reflect the changes that occur both on and off the platform. To be clear: the vast majority of content on YouTube does not violate the guidelines. But the platform still checks for gaps that may have opened up or hunt for emerging risks that test the policies in new ways.

As YouTube works to keep its policies evolving with the current landscape, the guiding focus is around one major goal: preventing egregious real-world harm. This doesn’t mean that it removes all offensive content from YouTube, and it is generally believed that open debate and free expression leads to better societal outcomes. However, the platform is careful about drawing the line around content that may cause egregious harm to its users or to the platform.

This can include physical harm. For example, when claims that linked 5G technology to the spread of COVID-19 resulted in damage to cell towers across the United Kingdom, YouTube moved quickly to make them violative. Or it could mean significant harm to democratic institutions, which is why it doesn’t allow claims that aim to mislead people about voting — including by promoting false information about the voting times, places or eligibility requirements.

YouTube also works closely with NGOs, academics, and relevant experts from all sides and different countries to inform this policy review. They help flag new concerns, or bring a deep understanding to complex topics that are prone to consistent change. For example, YouTube established its COVID-19 misinformation policy at the start of the pandemic alongside health authorities like the Center for Disease Control and World Health Organization. Later, as their guidance shifted to ease mask and social distancing restrictions, the platform updated its policies around content that questioned the efficacy of masks and social distancing.

Deciding on where to draw ‘the line’

Once an area has been identified where a policy update is needed, that’s where YouTube’s Trust & Safety team comes in to develop a tailored solution. It starts by assessing a few things. How commonly found is this specific type of harmful content on YouTube (and what’s its potential to grow)? And how is it managed under the current Community Guidelines?

Then the team watches dozens or even hundreds of videos to understand the implications of drawing different policy lines. Drawing a policy line is never about a single video; it’s about thinking through the impact on all videos, which would be removed and which could stay up under the new guideline. Following this comprehensive review, the team shares various options for policy lines, making sure to detail examples of videos that would be removed or approved for each (as well as different enforcement actions, like removal vs. age-restriction).

A top choice is selected from those draft options and then goes through further rounds of assessment. At this stage, the team is looking to understand whether the proposal can meaningfully achieve a few key goals:

  • Mitigate egregious real-world harm while balancing a desire for freedom of expression.
  • Allow for consistent enforcement by thousands of content moderators across the globe.

If the team is satisfied that it is hitting these targets, an executive group made up of leads across the company reviews the proposal. Final sign-off comes from the highest levels of leadership, including YouTube’s Chief Product Officer and CEO. If at any point there is consistent disagreement between teams about where the line is drawn, the policy is sent back to the drawing board.

Providing input on policy development and enforcement

Throughout the policy development process, YouTube partners closely with a range of established third-party experts on topics like hate speech or harassment. It also works with various government authorities on other important issues like violent extremism and child safety.

Experts help in forecasting how global events could cause harmful content to spread across the platform, including uncovering gaps in our systems that might be exploited by bad actors, or providing recommendations for new updates. And like with COVID-19, they provide input that helps the platform adapt policies in situations where guidance can change quickly.

These partnerships are also especially critical to support policy enforcement for regional issues, where language or cultural expertise is often needed to properly contextualise content. For example, YouTube worked closely with experts in 2021 during the coup d'état in Myanmar to identify cases where individuals were using speech to incite hatred and violence along ethno-religious lines. This allowed the platform to quickly remove the violative content.

Getting ahead of emerging issues

People often think about content moderation as reactive in nature — that YouTube only takes content down when it’s flagged by the systems or people. In reality, the bulk of the work focuses on the future. There’s a long process that’s designed to give the teams visibility into emerging issues before they reach, or become widespread on, the platform.

That valuable visibility is driven by the Intelligence Desk, a team within YouTube’s Trust & Safety organisation. These specialised analysts identify potentially violative trends — whether new vectors of misinformation or dangerous internet challenges — and the risks they pose. They’re also regularly monitoring ongoing threats like extremist conspiracy theories, both tracking their prevalence across media and evaluating how they morph over time.

These insights then feed into thinking through how current or future policies would manage these new threats. For example, based on evidence gathered by the Intelligence Desk, YouTube updated its hate and harassment policies to better combat harmful conspiracy theories on the platform.

Ensuring policies are enforced consistently

The implementation of a new policy is a joint effort between people and machine learning technology. In practice, that means in order for a policy to be successfully launched and enforced, people and machines need to work together to achieve consistently high levels of accuracy when reviewing content.

YouTube starts by giving its most experienced team of content moderators enforcement guidelines (detailed explanation of what makes content violative), and ask them to differentiate between violative and non-violative material. If the new guidelines allow them to achieve a very high level of accuracy, the platform expands the testing group to include hundreds of moderators across different backgrounds, languages and experience levels.

At this point, it begins to revise the guidelines so that they can be accurately interpreted across the larger, more diverse set of moderators. This process can take a few months, and is only complete once the group reaches a similarly high degree of accuracy. These findings then help train the machine learning technology to detect potentially violative content at scale. As it does with its content moderators, YouTube tests models to understand whether it has provided enough context for them to make accurate assessments about what to surface for people to review.

After this testing period, the new policy can finally launch. But the refinement continues in the months that follow. Every week, YouTube’s Trust & Safety leadership meet with quality assurance leads from across the globe (those responsible for overseeing content moderation teams) to discuss particularly thorny decisions and review the quality of our enforcement. If needed, guideline tweaks are then drafted to address gaps or to provide clarity for edge cases.

How do people and machines work together to enforce the policies?

Once models are trained to identify potentially violative content, the role of content moderators remains essential throughout the enforcement process. Machine learning identifies potentially violative content at scale and nominates for review content that may be against our Community Guidelines. Content moderators then help confirm or deny whether the content should be removed.

This collaborative approach helps improve the accuracy of YouTube’s models over time, as models continuously learn and adapt based on content moderator feedback. And it also means that the platform’s enforcement systems can manage the sheer scale of content that’s uploaded to YouTube (over 500 hours of content every minute), while still digging into the nuances that determine whether a piece of content is violative.

For example, a speech by Hilter at the Nuremberg rallies with no additional context may violate the hate speech policy. But if the same speech was included in a documentary that decried the actions of the Nazis, it would likely be allowed under YouTube’s EDSA guidelines. EDSA takes into account content where enough context is included for otherwise violative material, like an educational video or historical documentary.

This distinction may be more difficult for a model to recognise, while a content moderator can more easily spot the added context. This is one reason why enforcement is a fundamentally shared responsibility — and it underscores why human judgment will always be an important part of YouTube’s process. For most categories of potentially violative content on YouTube, a model simply flags content to a content moderator for review before any action may be taken.

Measuring success

YouTube is driven in all of its work to live up to its Community Guidelines and further its mission to allow new voices and communities to find a home on the platform. Success on this front is hard to pin down to a single metric, but YouTube is always listening to feedback from stakeholders and members of its community about the ways it can improve — and it continuously looks to provide more transparency into its systems and processes (including efforts like this blog).

To measure the effectiveness of its enforcement, YouTube releases a metric called the violative view rate, which looks at how many views on YouTube come from violative material. From July through September of this year, that number was 0.10% – 0.11%, which means that for every 10,000 views, between 10 and 11 were of content that violated the Community Guidelines.

YouTube also tracks the number of appeals submitted by creators in response to videos that are removed (an option available to any Creator on YouTube), as this helps the platform gain a clearer understanding about the accuracy of its systems. For example, during the same time period mentioned above, YouTube removed more than 5.6 million videos for violating its Community Guidelines and received roughly 271,000 removal appeals. Upon review, the platform reinstated about 29,000 appeals.

And while metrics like appeals, reinstatements, and the violative view rate don’t offer a perfect solution to understand consistency or accuracy, they’re still pivotal in benchmarking success on an ongoing basis.

Community Guidelines are concerned with language and expression — two things that, by their very nature, evolve over time. With that shifting landscape, YouTube intends to continue to regularly review its policy lines to make sure they’re drawn in the right place. And to keep its community informed, YouTube will be sharing further how it is adapting in the months ahead.

Media
@adgully

News in the domain of Advertising, Marketing, Media and Business of Entertainment