Things Everyone Should Understand About the Stanford AI Index Report

And my notes on why they’re important

Apr 23, 2024

The Stanford AI Index Report is an annual report published by the Stanford Institute for Human-Centered Artificial Intelligence (HAI). It's a comprehensive analysis of the state of artificial intelligence, covering a wide range of topics. You can access the full 502-page report here or the top 10 takeaways here. I discuss the takeaways in more detail below.

Purpose

Track AI Progress: The report meticulously tracks the development of AI across various domains, including technical capabilities, research trends, and real-world applications.
Inform Policy and Decision-Making: It provides crucial data and insights to help policymakers, business leaders, researchers, and the general public understand the complex and rapidly evolving AI landscape.
Highlight Ethical Considerations: The report emphasizes the need for responsible AI development and highlights ethical challenges that come with the technology.

Areas Covered

Technical AI Performance: Benchmarks and progress in areas like computer vision, natural language processing, robotics, and more.
Research and Development: Trends in publications, conferences, patents, and funding within the AI field.
Investment: Examines private and government investment in AI across different countries and sectors.
AI Adoption and Impact: How AI is being used in healthcare, finance, transportation, and other industries. It also looks at the economic impact.
Workforce and Education: Analysis of AI-related job trends, diversity initiatives, and educational initiatives.
Governance and Policy: Explores legislative efforts, regulatory frameworks, and ethical guidelines addressing AI.

Why It’s Important

The Stanford AI Index Report offers a data-driven, unbiased perspective on the advancements and challenges associated with AI. This helps various stakeholders make informed decisions about how to develop, deploy, and govern AI technologies.

The AI Index Report is written to be understood by anyone and should be read by everyone. It’s a quick way for anyone to get an overview of the important progress in AI over the previous year and an outlook toward where it’s headed.

Takeaways

1. AI beats humans on some tasks, but not on all.

AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.

With these advancements, we’ll see AI dominate certain jobs where they perform better and cost less than humans. Yes, this will cause some jobs to be replaced but the greater impact will be a shift in the job landscape. The role of most jobs will change as they’re augmented with AI. In tasks where AI isn’t able to achieve human-like performance, AI will still be used but it’ll have less of an impact for the time being.

2. Industry continues to dominate frontier AI research.

In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. There were also 21 notable models resulting from industry-academia collaborations in 2023, a new high.

Companies continue to take a greater interest in machine learning as more practical use cases become apparent. For many years, machine learning advancements were led by academia. Recent advancements have the potential for tremendous productivity gains and industry giants have devoted more time and resources to exploring these.

3. Frontier models get way more expensive.

According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.

A significant amount of capital is required to train state-of-the-art large language models from scratch. Some interesting observations:

As discussed in takeaway 2, companies with capital instead of research institutes are at the forefront of machine learning developments.
Open LLMs coming from these companies are important for smaller players to leverage state-of-the-art machine learning models without spending the time and money to train from scratch.
Comparing costs across models is interesting to compare resource efficiency and usage across companies.

4. The United States is ahead of China, the EU, and the UK as the leading source of top AI models.

In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union’s 21 and China’s 15.

The U.S. is currently dominating the AI race. This is likely due to less legislative hurdles and an environment where large companies are able to flourish. Highly capable AI will be huge for economic growth and a country’s place in the AI race is highly indicative of its economic future.

5. Robust and standardized evaluations for LLM responsibility are seriously lacking.

New research from the AI Index reveals a significant lack of standardization in responsible AI reporting. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. This practice complicates efforts to systematically compare the risks and limitations of top AI models.

I’ve recently written about AI benchmarks and why it’s important to understand them. Benchmarks are tests used to quantify performance. Small differences in the way benchmarks measure can lead to large differences in what they actually tell us. Using different benchmarks across AI industry leaders makes measuring and comparing AI responsibility difficult. The AI industry may need to find consensus on standard practices to make this easier.

6. Generative AI investment skyrockets.

Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds.

In the current stages of machine learning advancements, everyone is still figuring out how to practically utilize and profit from AI. Generative AI is a space where many productive applications have already been determined and the number of potential applications is growing quickly. The interest of even non-AI companies has paved the way for more investment in generative AI.

7. The data is in: AI makes workers more productive and leads to higher quality work.

In 2023, several studies assessed AI’s impact on labor, suggesting that AI enables workers to complete tasks more quickly and to improve the quality of their output. These studies also demonstrated AI’s potential to bridge the skill gap between low- and high-skilled workers. Still other studies caution that using AI without proper oversight can lead to diminished performance.

As stated in takeaway 1, AI is very good at some things but struggles with others. In the areas it is good at it does an excellent job of improving output. In the areas it is bad at it has had the opposite effect. This is expected, but highlights the issue of using AI without human oversight which has become a touchy subject between employees who fear their job may be replaced and upper management at companies looking to implement AI to increase profits.

8. Scientific progress accelerates even further, thanks to AI.

In 2022, AI began to advance scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications—from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of materials discovery.

2023 has proven the capabilities of AI in accelerating research applications, especially in non-AI research areas. A great example of this is AlphaFold. Research and development is at the core of great societal advancements, but is a notoriously slow and expensive process. Accelerating it with AI will have a huge societal benefit and further increase technological development and understanding.

9. The number of AI regulations in the United States sharply increases.

The number of AI-related regulations in the U.S. has risen significantly in the past year and over the last five years. In 2023, there were 25 AI-related regulations, up from just one in 2016. Last year alone, the total number of AI-related regulations grew by 56.3%.

In my note on takeaway 4, I mentioned a lack of regulations being one of the reasons the US has developed AI so much faster than the rest of the world. Regulations are important to protect individuals, but have the potential to unnecessarily stifle innovation. The key is to maximize the former without doing the latter. If regulation is too strict, the US may face a significant hurdle in a race it is currently winning.

10. People across the globe are more cognizant of AI’s potential impact—and more nervous.

A survey from Ipsos shows that, over the last year, the proportion of those who think AI will dramatically affect their lives in the next three to five years has increased from 60% to 66%. Moreover, 52% express nervousness toward AI products and services, marking a 13 percentage point rise from 2022. In America, Pew data suggests that 52% of Americans report feeling more concerned than excited about AI, rising from 38% in 2022.

AI will impact 100% of the population as it becomes the default way we solve complex problems. This means everyone will use AI in some capacity whether they know it or not. This is a good thing. AI has the potential to make everyone’s life better. Misconceptions are often spread to create fear and stifle progress. These misconceptions about AI are why I’ll be making YouTube videos in the near future to clear them up for everyone.

That’s the current state of AI. Let me know what you think and leave any questions in the comments below.

As always, you can reach me on X, LinkedIn, or Substack direct messages. You can also support Society’s Backend for $1/mo for the first year and get access to all my machine learning resources.