Human-centric tech will make AI faster and fairer. Here’s how.

(Credit: Unsplash)

This article is brought to you thanks to the collaboration of The European Sting with the World Economic Forum.

Author: Wilson Pang, Chief Technology Officer, Appen


  • We need human-centric technology that enhances productivity and drives greater ROI.
  • With ML-assisted technology embedded into the data labelling pipeline, you can reduce the time, money and people required.
  • There are many opportunities to prevent contributors from being biased by the model’s predictions.

Under traditional machine learning (ML) methods, humans perform the often time-consuming and expensive task of annotating each and every row of data needed; successful artificial intelligence (AI) models require thousands, if not millions, of units of accurately labeled training data.

As we evolve our approaches to AI, this level of manual effort becomes questionable.

Now that we have the foundation of multiple state-of-the-art pre-labeling models at our disposal, it’s imperative that we leverage them for process improvements in the end-to-end AI deployment cycle. These models include pre-labeling for autonomous vehicle image pixel labeling, pre-labeling for image and document transcription, pre-labeling for audio segmentation, and several other pre-label or classification models. In advancing our tooling, we need to invest in a certain type of human-centric technology: one that both enhances productivity and drives greater ROI.

Human-centred technology considers the operator to be an asset rather than an impediment. It recognises the value of the operator’s skill, knowledge, flexibility and creativity.

Our goal in optimizing human-centered technology should be two-fold: to create faster and more efficient AI pipelines without sacrificing quality, and to advance the fair treatment of contributors by reducing the human burden of labelling tasks which are often repetitive and mentally draining. Annotating videos often requires labeling the separate frames of the videos with very small changes to the annotations. For example, a video of cars driving down the road is broken into multiple frames and each vehicle needs to be labelled. These types of annotations would be exceedingly time consuming to do by hand given the number of frames which make up even a short video. By using machine learning, we can automate the annotation process, applying the annotation predictions immediately to the frames so the annotator can simply adjust as needed instead of having to create each annotation.

ML-assisted tools serve as the bedrock of our endeavour towards human-centric technology. With ML-assisted technology embedded into the data labelling pipeline, you can reduce the time, money and people required for this crucial step in model build.

It also provides the chance to automate and improve the quality and delivery of data annotation. In this approach (at Appen, we call it “smart labelling”), critical touchpoints exist before, during and after job completion.

Touch point one: before the job

Before you run an annotation job, you can leverage pre-trained or trainable models to provide an initial hypothesis for your data labels. Unlike manual labelling processes, your contributors will be checking the hypothesis for accuracy rather than adding a label from scratch.

For example, if you’re working on an image annotation job to identify cars on the road, you can use a pre-trained model to pre-classify those target objects or cars.

Various models can accomplish specific tasks, depending on your use case. These range from censoring explicit content to blurring out personal details and adding bounding boxes around objects. Using existing models to provide initial data labels saves time and money by automating a portion of the annotation process. The accuracy will depend on the model or combination of models that you select.

But how do we prevent contributors from being biased by the model’s predictions, you might ask?

In fact, we tested this by running large-scale A/B testing for several annotation projects and found quite the opposite to be true: pre-labelling data resulted in improved label quality. In other words, data that has the initial labels or annotations completed by an ML model before handing over to the contributor for final annotations resulted in higher quality labels than data that did not have initial labels.

In one image-pixel-labelling project for autonomous vehicles, using an ML model for initial labelling improved contributor productivity by 91.5% and annotation quality by 10% across all of our trials.

If your team is still concerned about bias, there are further opportunities for mitigation in the next two phases of the pipeline.

Touch point two: during the job

Once inside the job, you can leverage ML models to assist human judgments. As an example, if your job includes video annotation, a manual process might look like this: videos are split into frame-by-frame sequences and contributors label each target object in each frame.

With a standard frame rate of 24 frames per second, this labelling task becomes laborious and repetitive quickly. Using ML-assisted technology instead, the contributor can label the target object once and a model can track and predict its location in subsequent frames. Following the same example of cars on the road, the contributor would label each car in the first frame and the model would track its location to annotate the cars in the remaining frames.

Contributors then take on the role of reviewer for the remaining frames, making corrections as needed.

With help from ML-assisted technology during the job, contributors are equipped to work more quickly and with greater accuracy. Using this method can result in annotation speeds that are up to 100 times faster than manual methods, without sacrificing quality. The benefits extend to contributors as well: this method reduces cognitive strain, improving comfortability throughout the task.

Final touch point: after-work

After the model and contributor have made judgments on your data, you can enter the validation phase. In this step, you can use ML models to verify the judgments made and notify contributors if their input isn’t within the expected quality thresholds.

This approach has a couple of notable benefits. Notably, it removes any need for test questions or peer reviews and it also reduces the risk that you will end up paying for judgments that don’t fit your requirements. After model validation, the contributor can submit the job.

We need to invest not just in AI solutions, but also in improving the processes that support them. —Wilson Pang

If you have a text utterance project, for example, you can utilise ML-assisted validation tools combined with set indicators, such as coherence or language. The model will flag any data labels that don’t meet your accuracy requirements for these indicators.

A human annotator then reviews and corrects the labels. Appen tested ML-assisted validation tools in a text-utterance project involving the training of chatbots. We found a 35% reduction in error rates using real-time models.

‘It’s not just about AI but about better AI processes’

Combining machine learning with human effort in the form of human-centric technology is the way forward for AI innovation.

ML-assisted features in data annotation pipelines help both companies and contributors: companies can expend fewer resources in their launching of high-quality AI solutions and do so faster, and contributors can work on tasks that provide less mental strain and repetition. The latter is especially important in bolstering fair AI practices for all of the individuals who work on AI projects.

We need to invest not just in AI solutions, but also in improving the processes that support them. This way, we can evolve our approach to ethical AI and accelerate our ability to solve global issues with machine-driven solutions.

AI isn’t meant to rely on the machine or the human exclusively; rather, leveraging a combination of the two can enhance each other’s strengths and promote successful outcomes.

the sting Milestones

Featured Stings

Can we feed everyone without unleashing disaster? Read on

These campaigners want to give a quarter of the UK back to nature

How to build a more resilient and inclusive global system

Stopping antimicrobial resistance would cost just USD 2 per person a year

European Commission increases support for the EU’s beekeeping sector

More protection for our seas and oceans is needed, report finds

Which country offers the cheapest mobile data?

INTERVIEW: ‘Defend the people, not the States’, says outgoing UN human rights chief

10 ways central banks are experimenting with blockchain

Can the US-Iran rapprochement change the world?

DR Congo elections: ‘historic opportunity’ for ‘peaceful transfer of power’ says Security Council

What can be done to avoid the risk of being among the 7 million that will be killed by air pollution in 2020?

Is there a de facto impossibility for the Brexit to kick-start?

How trust and collaboration are key in India’s last mile response to the COVID-19 crisis

Investors must travel a winding road to net-zero. Here’s a map

Engaging women and girls in science ‘vital’ for Sustainable Development Goals

‘No steps taken’ so far to end Israel’s illegal settlement activity on Palestinian land – UN envoy

In visit to hurricane-ravaged Bahamas, UN chief calls for greater action to address climate change

Illegal fishing plagues the Pacific Ocean. Here’s how to end it

How AI and machine learning are helping to fight COVID-19

EU tells Britain stay in as long as you wish

Financing fossil fuels risks a repeat of the 2008 crash. Here’s why

Here are 4 tips for governing by design in the Fourth Industrial Revolution

How curiosity and globalization are driving a new approach to travel

Coronavirus (COVID-19): truth and myth on personal risk perception

The battle for the 2016 EU Budget to shake the Union; Commission and Parliament vs. Germany

Innovation can transform the way we solve the world’s water challenges

#WorldBicycleDay: 5 benefits of cycling

Missile strike kills at least 12 civilians, including children, in Syria’s Idlib: UN humanitarians

4 steps to developing responsible AI

Mental health and suicide prevention – What can be done to increase access to mental health services in my region?

UN chief outlines ‘intertwined challenges’ of climate change, ocean health facing Pacific nations on the ‘frontline’

New US President: MEPs hope for a new dawn in transatlantic ties

Desires for national independence in Europe bound by economic realities

European Union and African Union sign partnership to scale up preparedness for health emergencies

Yemen war: UN chief urges good faith as ‘milestone’ talks get underway in Sweden

Spring 2019 Standard Eurobarometer: Europeans upbeat about the state of the European Union – best results in 5 years

Coronavirus: Commission approves contract with CureVac to ensure access to a potential vaccine

Outbreak of COVID-19: The third wave and the people

A day in the life of a Venezuelan migrant in Boa Vista, Brazil

EU Copyright Directive: Google News threatens to leave Europe while media startups increasingly worry

3 ways to fight short-termism and relaunch Europe

Accountability in Sudan ‘crucial’ to avoid ‘further bloodshed’, says UN rights office

UN committed ‘to support the Libyan people’ as Guterres departs ‘with deep concern and a heavy heart’

Antarctica: the final coronavirus-free frontier. But will it stay that way?

Mario Draghi didn’t do it but Kim Jong-un did

UN chief welcomes G20 commitment to fight climate change

MEPs: Access to adequate housing should be a fundamental European right

More countries are making progress on corruption – but there’s much to be done, says a new report

Mountains matter, especially if you’re young, UN declares

EU food watchdog: more transparency, better risk prevention

Young activists share four ways to create a more inclusive world

The European Sting @ the European Business Summit 2014 – Where European Business and Politics shape the future

More than one million sexually transmitted infections occur every day: WHO

These countries spend the most on education

How a new approach to meat can help end hunger

MEPs cap prices of calls within EU and approve emergency alert system

Electronic cigarettes: is it really a safe alternative to smoking?

China confirms anti-state-subsidy investigation on EU wine imports

Century challenge: inclusion of immigrants in the health system

Here’s how we reboot digital trade for the 21st century

Britain and Germany change attitude towards the European Union

UN, global health agencies sound alarm on drug-resistant infections; new recommendations to reduce ‘staggering number’ of future deaths

Ten new migratory species protected under global wildlife agreement

More Stings?

Comments

  1. very nice information.

Speak your Mind Here

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s