Creative Innovation 2019 (Ci2019)

How Fast Is AI Progressing? Stanford’s New Report Card for Artificial Intelligence

Monday, 22 January 2018

When? This is probably the question that futurists, AI experts, and even people with a keen interest in technology dread the most. It has proved famously difficult to predict when new developments in AI will take place. The scientists at the Dartmouth Summer Research Project on Artificial Intelligence in 1956 thought that perhaps two months would be enough to make “significant advances” in a whole range of complex problems, including computers that can understand language, improve themselves, and even understand abstract concepts.

Sixty years later, and these problems are not yet solved. The AI Index, from Stanford, is an attempt to measure how much progress has been made in artificial intelligence.

The index adopts a unique approach, and tries to aggregate data across many regimes. It contains Volume of Activity metrics, which measure things like venture capital investment, attendance at academic conferences, published papers, and so on. The results are what you might expect: tenfold increases in academic activity since 1996, an explosive growth in startups focused around AI, and corresponding venture capital investment. The issue with this metric is that it measures AI hype as much as AI progress. The two might be correlated, but then again, they may not.

The index also scrapes data from the popular coding website Github, which hosts more source code than anyone in the world. They can track the amount of AI-related software people are creating, as well as the interest levels in popular machine learning packages like Tensorflow and Keras. The index also keeps track of the sentiment of news articles that mention AI: surprisingly, given concerns about the apocalypse and an employment crisis, those considered “positive” outweigh the “negative” by three to one.

But again, this could all just be a measure of AI enthusiasm in general.

No one would dispute the fact that we’re in an age of considerable AI hype, but the progress of AI is littered by booms and busts in hype, growth spurts that alternate with AI winters. So the AI Index attempts to track the progress of algorithms against a series of tasks. How well does computer vision perform at the Large Scale Visual Recognition challenge? (Superhuman at annotating images since 2015, but they still can’t answer questions about images very well, combining natural language processing and image recognition). Speech recognition on phone calls is almost at parity.

In other narrow fields, AIs are still catching up to humans. Translation might be good enough that you can usually get the gist of what’s being said, but still scores poorly on the BLEU metric for translation accuracy.

Measuring the performance of state-of-the-art AI systems on narrow tasks is useful and fairly easy to do. You can define a metric that’s simple to calculate, or devise a competition with a scoring system, and compare new software with old in a standardized way. Academics can always debate about the best method of assessing translation or natural language understanding. The Loebner prize, a simplified question-and-answer Turing Test, recently adopted Winograd Schema type questions, which rely on contextual understanding. AI has more difficulty with these.

Where the assessment really becomes difficult, though, is in trying to map these narrow-task performances onto general intelligence. This is hard because of a lack of understanding of our own intelligence. Computers are superhuman at chess, and now even a more complex game like Go. The braver predictors who came up with timelines thought AlphaGo’s success was faster than expected, but does this necessarily mean we’re closer to general intelligence than they thought?

Here is where it’s harder to track progress.

We can note the specialized performance of algorithms on tasks previously reserved for humans—for example, the index cites a Nature paper that shows AI can now predict skin cancer with more accuracy than dermatologists. We could even try to track one specific approach to general AI; for example, how many regions of the brain have been successfully simulated by a computer? Alternatively, we could simply keep track of the number of professions and professional tasks that can now be performed to an acceptable standard by AI.

“We are running a race, but we don’t know how to get to the endpoint, or how far we have to go.”

Progress in AI over the next few years is far more likely to resemble a gradual rising tide—as more and more tasks can be turned into algorithms and accomplished by software—rather than the tsunami of a sudden intelligence explosion or general intelligence breakthrough. Perhaps measuring the ability of an AI system to learn and adapt to the work routines of humans in office-based tasks could be possible.

The AI index doesn’t attempt to offer a timeline for general intelligence, as this is still too nebulous and confused a concept.

Michael Woodridge, head of Computer Science at the University of Oxford, notes, “The main reason general AI is not captured in the report is that neither I nor anyone else would know how to measure progress.” He is concerned about another AI winter, and overhyped “charlatans and snake-oil salesmen” exaggerating the progress that has been made.

A key concern that all the experts bring up is the ethics of artificial intelligence.

Of course, you don’t need general intelligence to have an impact on society; algorithms are already transforming our lives and the world around us. After all, why are Amazon, Google, and Facebook worth any money? The experts agree on the need for an index to measure the benefits of AI, the interactions between humans and AIs, and our ability to program values, ethics, and oversight into these systems.

Barbra Grosz of Harvard champions this view, saying, “It is important to take on the challenge of identifying success measures for AI systems by their impact on people’s lives.”

For those concerned about the AI employment apocalypse, tracking the use of AI in the fields considered most vulnerable (say, self-driving cars replacing taxi drivers) would be a good idea. Society’s flexibility for adapting to AI trends should be measured, too; are we providing people with enough educational opportunities to retrain? How about teaching them to work alongside the algorithms, treating them as tools rather than replacements? The experts also note that the data suffers from being US-centric.

We are running a race, but we don’t know how to get to the endpoint, or how far we have to go. We are judging by the scenery, and how far we’ve run already. For this reason, measuring progress is a daunting task that starts with defining progress. But the AI index, as an annual collection of relevant information, is a good start.

Read the original article.

"Without a doubt the best conference in Australia with a delightful mix of creativity and big thinking bringing together an eclectic group of industry, community and education leaders. Once again, I’ve come away enthused by new ideas, more aware of future trends, and heartened by the kindness of humanity. We all have a role to play and it should be a collective one." Kerry Anderson, Founder of Operation Next Gen

"Conferences are boring. Creative Innovation is more like a rodeo for ideas, and you are guaranteed to leave wanting to make the world better for everyone." Andrew Despi, A.kin

"Kudos to Tania for organising the the 2019 CI Conference in Melbourne - it is the best I have attended so far. All the speakers and sessions were very well-planned. I have attended many conferences and this is the first I did not have the 'luxury' to skip any sessions. Your effort in this space has truly made Melbourne a global innovation and entrepreneurship hub." Danny Ong, Deakin University

"I wanted to pass along how incredible I thought Creative Innovation was and how much your passion for life and for humanity was inspiring to all of us. It was a privilege to be included and one that I am very grateful for." John Pickering, Chief Behavioural Scientist

"Creative Innovation is a transformational platform built by changemakers for changemakers. It is an educational adventure showcasing the forefront of ideas internationally." Melissa Warner, Ci2019 Scholarship Winner & Education Officer, Mind Medicine Australia

"Creative Innovation 2019 was one of the most well-organised events I’ve ever taken part in. From beginning to the end, every detail had been planned through and taken care of. Secondly, are the production values. As a result, it was possible to attend all the sessions without suffering conference burn-out, happy in the knowledge that the next piece of content was guaranteed to delight and challenge the senses." Richard Claydon (Singapore)

"Well done on another successful Ci conference. Your energy, passion and sheer determination to ensure the conference delegates get a fantastic experience is admirable." VP People & Global Capability | P&GC, Woodside Energy


There are a host of incredible opportunities to partner with us on this world class innovation event. If you are interested in becoming a partner or creative collaborator for the upcoming conference, please contact:

Tania de Jong AM // Founder and Executive Producer
Tel: +61 (0)3 8679 6000

Alrick Pagnon
Tel: +61 (0)3 8679 6000

“Remarkable, mind blowing, brilliantly choreographed – easily the best conference ever and I’ve been to plenty. Truly wonderful and amazing.”

Jim Grant, Partner, Dattner Grant

“Thank you for a challenging and rewarding couple of days. The quality of the team you had at the conference was extraordinary! These events will change people and thereby the world.”

Craig Carolan, Director Private Wealth, ANZ

“I came to learn – I came away inspired! Best conference ever.”

Paul Duldig, University of Melbourne