Chatbots are on the rise. This approach accounts for their risks

(Credit: Unsplash)

This article is brought to you thanks to the collaboration of The European Sting with the World Economic Forum.

Author: Venkataraman Sundareswaran, Fellow, Mitsubishi Chemical Holdings


  • Regulations for AI are falling behind the exponential growth of the use of it.
  • Developing an internal governance framework for “high-risk chatbots” would take a company considerable time and resources.
  • A useful framework for chatbot use now exists that gives small and medium enterprises the opportunity to demonstrate leadership in the responsible use of AI.

Artificial Intelligence (AI) is expanding at breakneck pace into products and services that we use daily. There are well-known ethical concerns about the use of it, though. Perhaps the most widely discussed is facial recognition in public places and its impact on privacy.

An array of AI applications incite concerns regarding ethics, such as bias, transparency and explainability. Despite this, regulation of the use of AI is seriously lacking. The incremental and deliberate processes behind regulations are falling behind the exponential growth of the use of AI.

The European Commission stepped up to this challenge with its release this April of proposed new actions and rules for trustworthy AI. It was arguably the first comprehensive examination of the new norms needed to ensure that AI can be trusted.

How chatbots are set to chat even louder

Conversational AI is one of the most popular uses of this type of technology. It has found its way into smart speakers by Amazon and Google, and all smartphones have either Siri, Alexa or the Google voice assistant. It was estimated that the number of smart speaker owners in the US was set to reach nearly 94 million in 2021.

Conversational AI is also built into chatbots, which have also been widely deployed on websites, social media platforms and smartphone apps. The COVID-19 pandemic accelerated the use of chatbots in the healthcare sector.

The EU proposal is a risk-based approach, treating AI applications under four categories: unacceptable risk, high-risk, limited risk and minimal risk. For example, remote biometric identification (such as facial recognition) is classified as high-risk. This makes sense as there are several facial recognition concerns that can escalate very quickly to population-scale crises.

Chatbots used in healthcare must be classified as high-risk if concerns are to be adequately addressed.—Venkataraman Sundareswaran

However, the proposal cites chatbots as limited-risk AI systems and cautions that “users should be aware that they are interacting with a machine so they can take an informed decision to continue or step back.”

This binary decision (to use or not to use), while easy to suggest, is not always the best approach to regulation simply because the use of chatbots in society-critical areas such as healthcare results in many benefits. These include their ease of use, 24/7 access, low cost, reusability, consistency and ease of deployment.

Figure 1: Benefits of chatbots in healthcare Image: World Economic Forum

With all this in mind, the acceleration in the use of chatbots in healthcare will most likely continue beyond the pandemic, providing useful healthcare access and related services to vast populations. For example, customers of the National Health Service in the UK have access to a chatbot to face triage for “urgent but non-life-threatening conditions.”

Countries like Rwanda are seriously investing in this technology to extend healthcare access to remote populations. Such widespread availability and ease of use is only going to increase the uptake of this technology, rather than force people to make the kind of binary choice suggested in the EU proposal.

Bad chat happens

While there are many benefits of using chatbots in healthcare, there are some risks as well. For example, when a chatbot is used to triage patients, extreme care must be taken to make sure that the AI behind the triage decision has been trained on data that is accurately representative of the population that uses the chatbot.

This is not an easy thing to do, because securing historical data from the target population is not always possible for a number of reasons. These include local laws on the use of healthcare data, past data collection rigor and the costs of converting data.

Other concerns include the efficacy of chatbots in the target healthcare application, a lack of room for humans to intervene or override suggestions by chatbots, not enough transparency over the decisions made by chatbots, and questions about how well the data collected by the chatbot is secured. For a more detailed discussion on this, see this report.

Admittedly, the EU proposal does not go into the kind of detail we are presenting here and provides enough room to “reclassify” applications in higher-risk categories. Under the EU proposal, chatbots can be classified as “high risk” if they target “essential private and public services,” which presumably include healthcare. But chatbots used in healthcare must be classified as high-risk if concerns with this use, such as those mentioned above, are to be adequately addressed.

How to get a running start

As we make the suggestion to categorize chatbot use in healthcare as high-risk, we also recognise that most companies that are innovating in this space are small and medium enterprises that may not have the bandwidth or resources to develop the internal governance needed.

Even larger enterprises involved in the use of this technology such as hospitals, insurance companies, and government regulators, often lack the necessary understanding of the technology and its implications to be able to develop the governance actions needed at speed. Indeed, developing an internal governance framework takes considerable time and resources.

To address this challenge, we brought together stakeholders from all parts of the ecosystem – including chatbot developers, platform providers, medical professionals, academia, civil society and governments – to develop a governance framework, which is available for anyone to download and use.

The framework contains 10 principles (Figure 2 below), carefully curated by the multi-stakeholder participants. More importantly, the authors provide, for each of the 10 principles, a set of actions that can be used to “operationalize” the principles.

Figyre 2: Out-of-the box principles available in the Chatbots RESET framework Image: World Economic Forum

By incorporating the actions in their chatbot development or deployment plan, the various players in the ecosystem (developers, providers and regulators) can quickly set themselves on a path to compliance with upcoming regulations, such as that from EU.

Finally, the use of the framework gives small and medium enterprises the opportunity to demonstrate leadership in responsible use of AI (chatbots) in healthcare, besides providing them a running start.

the sting Milestones

Featured Stings

Can we feed everyone without unleashing disaster? Read on

These campaigners want to give a quarter of the UK back to nature

How to build a more resilient and inclusive global system

Stopping antimicrobial resistance would cost just USD 2 per person a year

Italy’s dilemma after Merkel-Hollande agreed loose banking union

Nokia wins Commission’s approval for Alcatel-Lucent acquisition: a new way for antitrust cases?

9 steps to bridging the net-zero funding gap

Innovating together: connectivity that matters

Quality coffee can boost local economies and benefit farmers – here’s how

What has changed in the French politico-economic horizon

Ambitions are affordable for Asia and the Pacific

Here’s how drone delivery will change the face of global logistics

European Youth Forum welcomes strong stance on human rights in State of the Union

UN food agency appeals for access to key storage facility amid fight for Hudaydah

Millions of young people need better job skills. Here’s how businesses can help

‘Leaders who sanction hate speech’ encourage citizens to do likewise, UN communications chief tells Holocaust remembrance event

4 ways to deliver social justice during the COVID-19 recovery

‘Global trust’ declining, ‘our world needs stepped-up global leadership’

Occupational safety and health in a changing world of work

Do not take the EP’s consent on MFF for granted, says Budget Committee Chair

Global spotlight on world drug problem ‘is personal’ for many families, says UN chief

Parliament gives green light to EU-Singapore trade and investment protection deals

UN global education envoy urges new funding for ‘lost generation’ of children forced out of classrooms by conflict

Our Amazon is disappearing in ashes

FROM THE FIELD: ‘Miraculous’ music made by hearing-impaired children

Bias in AI is a real problem. Here’s what we should do about it

Three ways China can make the New Silk Road sustainable

Why skills are keeping CEOs awake at night

EU mobilises emergency assistance following floods in Ukraine

How quantum computing could beat climate change

4 ways to build a net-zero economy by 2050

Coronavirus: EU makes available additional humanitarian funding of €41 million to fight the pandemic

Technology companies have power. They must assume responsibility

EU Copyright Directive: Will US tech giants comply or ditch the EU market?

To Fight the Pandemic, Put Trust and Cooperation Before Politics

The EU slowly exits from “Excessive Deficit Procedure” and hopefully from ‘Excessive Austerity Procedure’ too

Things are bad and getting worse for South Africa. Or are they?

200 women call on tech giants to prioritize online safety. Here’s how

Aid teams respond to escalating southwest Syria conflict: 750,000 civilians are at risk

Libya: Attack on foreign ministry, an attack on all Libyans, stresses UN envoy

Mobility package: Parliament adopts position on overhaul of road transport rules

2020 EU Budget: Performance report highlights swift and global EU response to the coronavirus crisis while supporting key EU priorities

European Youth Forum welcomes establishment of new Youth Intergroup in the European Parliament

Take-home pay growing at lowest level since 2008, as gender-gap persists: UN labour agency

FROM THE FIELD: A UN peacekeepers-eye view of DR Congo

EU-Ukraine Summit: Moving Forward Together

MEPs oppose EU Commission plans to authorise three herbicide-resistant GMOs

Afghanistan: lead MEPs demand safe departure of EU nationals and Afghan partners and urgent tackling of humanitarian crisis

Here’s how sustainable aviation fuel can take off in Europe

A Sting Exclusive: “EU’s Sustainable Finance Action Plan – Laying down the foundations for a Greener Financial System”, by European Commission’s Vice-President Dombrovskis

4 myths about corruption

Climate change and health: public health awareness in an international framework

Wednesday’s Daily Brief: Diplomacy for Peace Day, #VaccinesWork, the cost of war on Afghans, tech and well-being

VAT: New e-commerce rules in the EU will simplify life for traders and introduce more transparency for consumers

Impressions of China

Five things everybody needs to know about the future of Journalism

OECD Secretary-General Gurría welcomes announcement of new trade agreement between the US, Mexico and Canada

A year on from Yemen talks breakthrough, top UN Envoy hails ‘shift’ towards peace, despite setbacks

LGBTQI+ and medicine, in the Land of the Pure

Protecting the ocean is key to fighting climate change

Asian and Pacific economies: decreases in tax revenue highlight need to broaden tax bases

New UN Global Climate report ‘another strong wake-up call’ over global warming: Guterres

This is how to make driving an EV more accessible and affordable

Banking on sunshine: world added far more solar than fossil fuel power generating capacity in 2017

More Stings?

Speak your Mind Here

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: