Chatbots are on the rise. This approach accounts for their risks

(Credit: Unsplash)

This article is brought to you thanks to the collaboration of The European Sting with the World Economic Forum.

Author: Venkataraman Sundareswaran, Fellow, Mitsubishi Chemical Holdings


  • Regulations for AI are falling behind the exponential growth of the use of it.
  • Developing an internal governance framework for “high-risk chatbots” would take a company considerable time and resources.
  • A useful framework for chatbot use now exists that gives small and medium enterprises the opportunity to demonstrate leadership in the responsible use of AI.

Artificial Intelligence (AI) is expanding at breakneck pace into products and services that we use daily. There are well-known ethical concerns about the use of it, though. Perhaps the most widely discussed is facial recognition in public places and its impact on privacy.

An array of AI applications incite concerns regarding ethics, such as bias, transparency and explainability. Despite this, regulation of the use of AI is seriously lacking. The incremental and deliberate processes behind regulations are falling behind the exponential growth of the use of AI.

The European Commission stepped up to this challenge with its release this April of proposed new actions and rules for trustworthy AI. It was arguably the first comprehensive examination of the new norms needed to ensure that AI can be trusted.

How chatbots are set to chat even louder

Conversational AI is one of the most popular uses of this type of technology. It has found its way into smart speakers by Amazon and Google, and all smartphones have either Siri, Alexa or the Google voice assistant. It was estimated that the number of smart speaker owners in the US was set to reach nearly 94 million in 2021.

Conversational AI is also built into chatbots, which have also been widely deployed on websites, social media platforms and smartphone apps. The COVID-19 pandemic accelerated the use of chatbots in the healthcare sector.

The EU proposal is a risk-based approach, treating AI applications under four categories: unacceptable risk, high-risk, limited risk and minimal risk. For example, remote biometric identification (such as facial recognition) is classified as high-risk. This makes sense as there are several facial recognition concerns that can escalate very quickly to population-scale crises.

Chatbots used in healthcare must be classified as high-risk if concerns are to be adequately addressed.—Venkataraman Sundareswaran

However, the proposal cites chatbots as limited-risk AI systems and cautions that “users should be aware that they are interacting with a machine so they can take an informed decision to continue or step back.”

This binary decision (to use or not to use), while easy to suggest, is not always the best approach to regulation simply because the use of chatbots in society-critical areas such as healthcare results in many benefits. These include their ease of use, 24/7 access, low cost, reusability, consistency and ease of deployment.

Figure 1: Benefits of chatbots in healthcare Image: World Economic Forum

With all this in mind, the acceleration in the use of chatbots in healthcare will most likely continue beyond the pandemic, providing useful healthcare access and related services to vast populations. For example, customers of the National Health Service in the UK have access to a chatbot to face triage for “urgent but non-life-threatening conditions.”

Countries like Rwanda are seriously investing in this technology to extend healthcare access to remote populations. Such widespread availability and ease of use is only going to increase the uptake of this technology, rather than force people to make the kind of binary choice suggested in the EU proposal.

Bad chat happens

While there are many benefits of using chatbots in healthcare, there are some risks as well. For example, when a chatbot is used to triage patients, extreme care must be taken to make sure that the AI behind the triage decision has been trained on data that is accurately representative of the population that uses the chatbot.

This is not an easy thing to do, because securing historical data from the target population is not always possible for a number of reasons. These include local laws on the use of healthcare data, past data collection rigor and the costs of converting data.

Other concerns include the efficacy of chatbots in the target healthcare application, a lack of room for humans to intervene or override suggestions by chatbots, not enough transparency over the decisions made by chatbots, and questions about how well the data collected by the chatbot is secured. For a more detailed discussion on this, see this report.

Admittedly, the EU proposal does not go into the kind of detail we are presenting here and provides enough room to “reclassify” applications in higher-risk categories. Under the EU proposal, chatbots can be classified as “high risk” if they target “essential private and public services,” which presumably include healthcare. But chatbots used in healthcare must be classified as high-risk if concerns with this use, such as those mentioned above, are to be adequately addressed.

How to get a running start

As we make the suggestion to categorize chatbot use in healthcare as high-risk, we also recognise that most companies that are innovating in this space are small and medium enterprises that may not have the bandwidth or resources to develop the internal governance needed.

Even larger enterprises involved in the use of this technology such as hospitals, insurance companies, and government regulators, often lack the necessary understanding of the technology and its implications to be able to develop the governance actions needed at speed. Indeed, developing an internal governance framework takes considerable time and resources.

To address this challenge, we brought together stakeholders from all parts of the ecosystem – including chatbot developers, platform providers, medical professionals, academia, civil society and governments – to develop a governance framework, which is available for anyone to download and use.

The framework contains 10 principles (Figure 2 below), carefully curated by the multi-stakeholder participants. More importantly, the authors provide, for each of the 10 principles, a set of actions that can be used to “operationalize” the principles.

Figyre 2: Out-of-the box principles available in the Chatbots RESET framework Image: World Economic Forum

By incorporating the actions in their chatbot development or deployment plan, the various players in the ecosystem (developers, providers and regulators) can quickly set themselves on a path to compliance with upcoming regulations, such as that from EU.

Finally, the use of the framework gives small and medium enterprises the opportunity to demonstrate leadership in responsible use of AI (chatbots) in healthcare, besides providing them a running start.

the sting Milestones

Featured Stings

Can we feed everyone without unleashing disaster? Read on

These campaigners want to give a quarter of the UK back to nature

How to build a more resilient and inclusive global system

Stopping antimicrobial resistance would cost just USD 2 per person a year

Here’s how to achieve growth in the Middle East and North Africa

UN Security Council condemns Taliban offensive as a blow against ‘sustainable peace’

Facebook and Google to treat Europe as the 51st State of the USA

European Solidarity Corps 2021-2027: First call for proposals to support youth volunteering activities

How music can help children with autism connect

To Bing or Not to Bing? That is the question

Gender equality: an issue much talked about but less acted upon

Terrorism and migrants: the two awful nightmares for Europe and Germany in 2016

Portugal can use its economic recovery to build up resilience

Pharmaceuticals spend millions to push TTIP while consumer groups spend peanuts

The EU Parliament slams Commission on economic governance

An economist explains the pros and cons of globalization

Protecting the front line: the healthcare of health professionals

State aid: Commission approves French scheme deferring payment by airlines of certain taxes to mitigate economic impact of coronavirus outbreak

Low productivity jobs continue to drive employment growth

“Fortress Europe”, “Pegida” and its laughing stocks

New challenge: Not going through “burnout” in times of quarantine

Mental and comportamental health in the pandemic context

An Easter Special: Social protection of migrants in Europe as seen through the eyes of European youth

Pharmaceuticals conceal drug side effects with the EU’s Court blessing

European Citizens’ Initiative: Commission registers ‘Mandatory food labelling Non-Vegetarian / Vegetarian / Vegan’ initiative’

Greenery: the miracle cure for urban living

Italy should boost investment in training for the future of work

Sustainable Finance: Commission welcomes the adoption by the European Parliament of the Taxonomy Regulation

Macro-Financial Assistance: Europe’s way to control Ukraine?

Companies can help build a more inclusive world. Here’s how

Joris in Indonesia

5 steps businesses can take to protect air quality after COVID-19

5 charts that show renewable energy’s latest milestone

Devastating storms like Hurricane Florence ‘unusual this far north’: UN weather agency

Inflammation is the fuel that feeds the cancer flame. So how do we fight back?

Digital democracy: a Swiss view on digital trust

Four in five adolescents failing to exercise for even 60 minutes a day, UN health agency warns

Gaza: deadly violence continues to escalate, top UN officials work to restore calm

This is Germany’s $45 billion, 18-year plan to move away from coal

‘Time is of the essence’ for refugees on Greek islands – UN agency

Huawei answers allegations about its selling prices

Break taboo around menstruation, act to end ‘disempowering’ discrimination, say UN experts

Can the whole world live in peace?

From Russia with love: Brussels and Moscow close to an agreement on Ukraine’s gas supplies

Innovations for Content Professionals at the DCX exhibition 2018 in Berlin, in association with The European Sting

EU prolongs economic sanctions on Russia by six months

Business is a crucial partner in solving the mental health challenge

UN chief welcomes event reuniting families on the Korean Peninsula

Rule of Law: Commission launches infringement procedure to protect the independence of the Polish Supreme Court

Auditors say EU spending delivers limited value for money but the timing of their report poses questions

Ebola Outbreak in Democratic Republic Congo is ‘largely contained’: WHO

Mergers: Commission refers acquisition of newly created joint venture by Telefónica and Liberty Global to the UK competition authority

Brexit: visa-free access to the EU for UK nationals and to the UK for Europeans

Statement following the European Medicines Agency review of the COVID-19 vaccine AstraZeneca

“Decisions taken in the coming weeks will shape Europe’s experience of the internet”, Joe Mcnamee from EDRi says live from European Business Summit 2015

We must rethink and repurpose cybersecurity for the COVID-19 era

Free and secure access needed in DR Congo conflict zone to tackle Ebola – WHO

Here’s what keeps CEOs awake at night (and why it might be bad news for your next job)

President Ursula von der Leyen welcomes the first official submission of a recovery and resilience plan by Portugal

THE COMMITTEES: ‘All roads lead to the Fifth’

EU summit: step up work for recovery, and update migration and asylum system

The Bavarians threaten Berlin and Brussels with immigration crisis

EU lawmakers vote to reintroduce visas for Americans over “reciprocity principle”

EU helps tackle air pollution in Kosovo with €76.4 million

More Stings?

Speak your Mind Here

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s