MaryGrace Stephenson

On December 12, 2024, the third installment of our AGI Icons series was held in Vancouver. The event featured Jeff Dean, Chief Scientist of Google DeepMind and Google Research, and Jonathan Siddharth, Co-Founder and CEO of Turing. 

Turing’s AGI Icons series is designed to bring together AI’s leading minds and host deep dialogues on the biggest barriers and achievements driving innovation. The conversation with Dean and Siddharth provided an exclusive look into Google’s latest model, Gemini 2.0 Flash, and the ways it’s poised to accelerate AGI advancements.

Dean’s journey with neural networks

Siddharth dove into the heart of the conversation: Dean’s journey with neural networks that culminated in Gemini 2.0.

“You’ve had a fascinating history with neural networks that goes quite far back. Can you talk us through what got you [started] on your journey?”

Jeff Dean’s fascination with neural networks began during his undergraduate studies at the University of Minnesota in 1990. 

“I was intrigued because they seemed like this really nice general learning mechanism that could solve problems that we couldn’t solve any other way. I felt like if we could just get more computation, we could train bigger and bigger neural networks.”

Inspired by his thesis, Dean undertook a project exploring parallel training strategies on a 32-processor hypercube machine. 

“I was naïve,” Dean admitted. “I thought training neural networks on 32 processors would revolutionize everything. What we really needed was a million times more compute power.”

Though the technology of the era was insufficient, these experiments laid the foundation for his future work in model and data parallelism,

Although he admits his efforts “weren’t very practical” for real-world applications at the time, he kept the concepts “in the back of my head.”

Fast-forward to 2011, when Dean was at Google. “I bumped into Andrew Ng because he had signed on as a one-day-a-week consultant.” 

When asked what he was doing there, Andrew replied “I don’t know yet, but my students at Stanford are starting to play with neural networks and getting good results.” This immediately sparked Dean’s interest. 

“That was when I started the Google Brain effort. And one of our first things we did was build a parallel and distributed system for training neural networks.”

Partnering with Andrew Ng, Dean’s focus on large-scale neural network training using Google’s infrastructure marked a turning point in AI history.

Neural networks at scale

Dean recounted how Google Brain tackled neural networks at scale, despite the limitations of hardware at the time.

“We didn’t have GPUs in our data centers, so we had lots and lots of CPUs,” Dean explained. “We used 2,000 computers, 16,000 cores to train interesting unsupervised computer vision models, speech recognition acoustic models, word embedding systems, and eventually Long short-term memory (LSTM) for sequence-to-sequence models”

The foundational principle was simple: “We want to train very large models using lots and lots of compute and more data.”

This mantra resonated with Siddharth, whose mission is to solve the next bottleneck in AI advancement – moving beyond compute with expert data and human intelligence. “Bigger model, more data, better results.”

Dean agreed. That mantra became the bedrock of modern AI advancements.

“I feel like if the scaling laws could be represented by a person, it would probably be you,” Jonathan joked. “You saw the power of neural networks early, the power of scaling up compute early, distributed training, DisBelief…”

Dean laughed, recalling the origins of DisBelief, an internal distributed training system at Google. “Yeah, and it was also a little bit of a double meaning because some people I talked to within Google were very skeptical that neural networks would work.” 

“So I said, ‘Ah, we’ll call it DisBelief, and we’ll show them.’”

Jonathan nodded, reminiscing about the era when neural networks weren’t always the go-to choice.

“When I was taking Andrew Ng’s class at Stanford in the mid-2000s, the belief was that for text classification, you probably wanted to use a support vector machine.” Siddharth continued, “Neural networks were—well, there was that phase, right?”

The genesis of Google DeepMind

The discussion then shifted to DeepMind and how their work complemented Google Brain’s efforts and culminated in the creation of Gemini. 

Initially, the two teams operated independently. Dean explained, “Within the Brain team, we focused on large-scale training, scaling things up, attacking a bunch of practical problems in vision, speech recognition, and language. DeepMind had focused a bit more on much smaller-scale models, reinforcement learning as a way of learning to do things that you couldn’t easily do with supervised learning.”

As their research agendas converged, merging efforts became the logical step. Dean’s leadership helped unify these teams into Google DeepMind, leveraging their combined strengths to develop state-of-the-art multimodal models.

“Instead of multiple independent efforts where we fragmented our compute and ideas, we decided to work on one unified model that’s multimodal. By combining our efforts, we’ve achieved something greater than the sum of its parts.”

Introducing Gemini 2.0 Flash

This unified approach became Gemini, Google’s flagship multimodal AI model. Siddharth then used the opportunity to transition into the Gemini announcement, asking, “We heard about Gemini 2.0 Flash. Is it number one on SWE-bench verified now?”

“Yeah, I believe it is,” Dean nodded. “We have this expression around our coffee machine in the micro kitchen at work: Such a good model.”

“So what’s new in Gemini 2.0 Flash?” Siddharth probed on behalf of the audience. 

“One is we’re announcing the 2.0 series of Gemini and coming out with the Flash model that people can use today,” he said. “The Gemini 2.0 Flash model outperforms on a bunch of academic benchmarks, at the latency and speed characteristics of 1.5 Flash but with improved quality.”

Dean then highlighted several innovations that set Gemini 2.0 apart:

• Multimodal capabilities – Gemini 2.0 can process and generate native audio, video, and end-to-end images within a single model, enabling seamless end-to-end learning.
• Agentic workflows – Features like Project Mariner allow Gemini to execute complex tasks, such as booking flights or navigating websites, while maintaining user control through safety guardrails.
• Real-world applications – The model brings practical improvements over its predecessors with tools like Project Astra, a personal assistant that integrates seamlessly into daily life, and assists with real-world software engineering tasks like code generation and debugging.

In essence: “These models can take in what you’re sensing and augment your knowledge in a natural and fluid way,” Dean explained.

Scaling the future of AI

The conversation then shifted to broader AI challenges and opportunities. Looking ahead, Dean outlined the key frontiers for AI. “You’ll see more interleaved processing and models thinking about what to do next, trying different approaches and learning from failed attempts,” he said. 

He also underscored the need for modular and sparse model architectures, advocating for designs inspired by the brain’s organic and efficient structure. Ultra-sparse models, which activate only the necessary parameters during inference, represent a path to greater efficiency.

Siddharth and Dean also discussed the role of software engineering in advancing AI. “As AI gets better at coding, what’s your advice for engineers?” posed Siddharth. 

Dean’s response was reassuring: “Some people are worried that AI is going to write software and there won’t be as much need for software engineers. I don’t subscribe to that at all. If anything, AI will enable engineers to be much more productive and spend time solving higher-order problems.”

Ethical AI and the value of expert human data

The conversation moved to the topic of ethical AI, focusing on the value of human-generated data and the need for fairness, transparency, and proper recognition of contributors.

Audience members raised questions about the implications of using human data to train LLMs  and the challenge of ensuring equitable value distribution. Dean emphasized the importance of balancing innovation with responsibility:

“We need community-driven solutions to ensure fairness,” he said. He noted Google’s opt-out policy as a step in the right direction, but stressed that the industry must do more to balance innovation with responsibility. He also highlighted the need for community-driven solutions to recognize and reward data contributions.

One audience member asked: “There must be a lot of data folks working hard to prepare the data for models like Gemini. How do we ensure they get the recognition they deserve in this large community?” 

Dean’s response underscored the collective nature of building cutting-edge AI models like Gemini: “The Gemini models to-date are the efforts of an amazing team at Google from all across Google DeepMind, Google Research, and our infrastructure teams. Some work on data, others on model architecture, and many others on infrastructure software or post-training processes—all of which are crucial.” 

He added: “It’s important to recognize that creating these models takes a collective effort, and we also rely on external partners like Turing for data curation and labeling.” 

Siddharth emphasized this point further, noting, “Data is the foundation of everything we do, and it’s important to ensure its value is shared equitably.”

Looking ahead–the future of AI

In a surprise announcement, Siddharth announced the creation of the Jeff Dean Award, which will recognize engineering excellence among Turing’s global community of developers. 

“This award will spotlight those pushing the boundaries of what’s possible,” Siddharth explained.This honor celebrates engineers who have made significant contributions to advancing AI and software development.

“It’s an honor to be part of a community that’s driving meaningful change.” Dean expressed his gratitude, emphasizing the importance of fostering a worldwide community of innovators. “I love the ethos of bringing people together all over the world to work on awesome stuff.”

As Siddharth noted, “You have such a unique vantage point to see where AI is headed. Where do you see the AI trajectory for the future that you can predict?”

Dean reflected on the extraordinary potential of AI.

Dean’s vision for the future of AI was ambitious yet pragmatic, painting a picture of systems that are increasingly capable, efficient, and collaborative.

“You’ll see a lot more interleaved processing,” Dean began, “where models think about what to do next, try different approaches, and backtrack when something doesn’t work. Achieving robust multi-step reasoning is one of the big challenges ahead. Right now, models might break down a task into five steps and succeed 75% of the time. But what we really want is for systems to break a task into 50 steps and succeed 90% of the time.”

He emphasized that building this level of reliability will unlock entirely new possibilities. “The ability to handle truly complex, multi-step tasks with consistent success—that’s when these systems will feel transformative in their problem-solving abilities.”

Dean concluded with a call for continued collaboration between engineers, researchers, and the broader AI community. “We’re just scratching the surface of what these systems can do,” he said. “The future lies in enabling people to accomplish complex tasks with ease and creativity. AI will augment human intelligence, helping us solve problems we’ve only dreamed of.”

In Summary

The conversation encapsulated the ethos of Turing’s AGI Icons series—bringing together thought leaders to illuminate the challenges and opportunities shaping AI. 

As Siddharth aptly summarized: “This is just the beginning. The best is yet to come.”

Model evaluation has emerged as a critical tool to enhance LLM’s performance and ROI. By systematically identifying inefficiencies, uncovering growth opportunities, and providing predictive analytics, model evaluation can significantly impact a model’s performance, keeping it on track with its intended purpose and improving effectiveness and reliability.

However, a one-size-fits-all approach to model evaluation is ineffective. Evaluations must consider diverse applications, tailored performance metrics, adaptability, scalability, ethical considerations, and real-world impact. Tailoring model evaluation to your business needs ensures you get the most value from your AI model, keeping it accurate, efficient, and reliable.

5 Misconceptions About LLM Evaluation

Model evaluation is crucial for ensuring AI models are accurate, efficient, and reliable. Yet, many companies fail to prioritize it, overlook its importance, or struggle to implement it effectively—oftentimes due to some common misconceptions.

Misconception #1: Costly investment vs actual ROI

Many believe that model evaluation is prohibitively expensive. However, thorough evaluations can lead to significant long-term cost savings by preventing errors and reducing inefficiencies, ultimately optimizing resources. These savings are often difficult to quantify because they result from the elimination of risk.

Consider the 1998 Mars Climate Orbiter mission, which failed to conduct proper evaluation before launching their spacecraft. The lack of assessment created cost savings upfront, but missed a critical unit conversion error—the navigation software used imperial units while the ground team used metric units. This oversight wasn’t identified before deployment, leading to the $125 million spacecraft’s loss. 

Misconception #2: Evaluation uses universal frameworks

Not all evaluation frameworks work for every model. Different models require tailored frameworks to capture critical nuances, with application-specific metrics and benchmarks providing the most accurate assessments.

Misconception #3: Model evaluation is a one-time process

Another misconception is that model evaluation is a one-time process. Effective model evaluation is iterative, adapting to new data and evolving requirements, ensuring scalability and continuous improvement.

Misconception 4: Evaluation metrics are only about accuracy / factuality

While accuracy is important, effective evaluation encompasses a variety of metrics, including precision, F1 score, computational efficiency, and user satisfaction, providing a holistic view of model performance.

Misconception #5: Evaluation is for regulatory compliance

It’s a common belief that evaluations are only necessary for regulatory compliance. In reality, evaluations validate a model’s real-world value and feasibility before committing extensive resources, refining the model to better meet business needs.

Defining Your Model Objectives 

To choose the right evaluation framework, start by clearly defining your model’s objectives. Understanding the primary purpose of your LLM will guide you in selecting the most appropriate evaluation criteria. 

What do you need your LLM to do?  

Define your business goals and how an LLM can support these objectives. Identify key areas where AI can provide value or solve critical problems. Then identify the specific tasks and functions you want the LLM to perform, such as:

• Automated response systems: Virtual assistants for customer service, support, and troubleshooting
• Content Generation and Summarization: Creating marketing copy, blog posts, social media content, and summarizing documents
• Code Generation and Software Development: Writing, debugging, and automating coding tasks
• Data Analysis, Forecasting, and Insights: Uncovering trends and forecasting future insights

How will you measure success? 

Once you have determined the purpose of your LLM, the next step is to identify key performance indicators (KPIs) that matter for your application. 

KPIs could include accuracy, fluency, coherence, relevance, precision, recall, computational efficiency, scalability, robustness, user interaction, compliance, security, ethical reasoning, and ROI. Setting clear performance goals will help you measure the success of your model and ensure it meets your business needs.

Evaluation Frameworks & Strategies

Based on the identified objectives and KPIs, select the appropriate evaluation frameworks and tools that align with your specific needs. 

Intrinsic Evaluation Frameworks 

Focus on the immediate output quality of the model, such as text coherence and accuracy. Automated testing tools like Weights & Biases, Azure AI Studio, and LangSmith can streamline the evaluation process.

• Automated Testing: Tools like Weights & Biases, Azure AI Studio, and LangSmith automate testing processes, ensuring consistent and thorough evaluations.
• Continuous Monitoring: Implementing continuous monitoring helps track model performance over time.
• Benchmarking: Use benchmarks such as BLEU, ROUGE, and F1 scores to measure text coherence and accuracy.

Extrinsic Evaluation Frameworks 

Concentrate on the model’s impact in real-world applications. Metrics-based evaluation, task-specific evaluation, human evaluation, user feedback, and robustness checks ensure comprehensive assessment.

• Metrics-based evaluation: Assess models using specific metrics tailored to the application.
• Task-specific evaluation: Evaluate how well the model performs specific tasks relevant to its use case.
• Human evaluation: Involve human evaluators to provide qualitative insights into model performance.
• User feedback: Gather feedback from end-users to understand the model’s impact and usability.
• Cross-validation and holdout validation: Use these techniques to ensure your model generalizes well to new data.
• Robustness checks and fairness testing: Ensure the model performs reliably under various conditions and is free from bias.

Use Established Benchmarks & Evaluation Tools 

Utilize well-known benchmarks and datasets to ensure comparability with other models. Benchmarking provides a standardized way to measure model performance across various tasks, offering insights into how your model stacks up against the competition.

• GLUE Benchmark: This benchmark is designed for natural language understanding tasks and includes a diverse set of challenges that test different aspects of language comprehension.
• SuperGLUE: Building on the GLUE benchmark, SuperGLUE introduces more complex tasks and comprehensive human baselines, pushing models to their limits and providing a more rigorous assessment.
• HellaSwag: This benchmark focuses on sentence completion tasks, evaluating the model’s ability to understand and predict the next part of a text sequence accurately.
• TruthfulQA: An essential benchmark for measuring the truthfulness of model responses, ensuring that the information generated by the model is accurate and reliable.
• MMLU: The Massive Multitask Language Understanding benchmark evaluates a model’s ability to handle multiple tasks simultaneously, testing its versatility and robustness across different domains.

By utilizing these established benchmarks, you can ensure that your model’s performance is assessed against high standards, providing a clear picture of its strengths and weaknesses.

Best Practices and Evaluation Use Cases

Continuous improvement and fine-tuning are vital for maintaining and enhancing the performance of LLMs over time. Here’s how to ensure your models stay at the top of their game:

• Broader Language Testing: Extend your evaluations to include more programming languages beyond Python. This ensures your model’s versatility and ability to handle diverse linguistic challenges.
• Continuous Improvement: Regularly update and test your models to refine prompt strategies and enhance LLM capabilities. This iterative process helps in identifying and fixing issues proactively.
• Continuous Monitoring: Regularly track model performance and retrain as needed based on new data and changing conditions.
• Feedback Loops: Incorporate user feedback into the evaluation process. Understanding how real users interact with your models and what they need can help align model outputs with user expectations, ensuring higher satisfaction and effectiveness.
• Performance Tracking: Implement robust performance tracking systems to gather real-time data on how your models perform in various scenarios. This data is crucial for making informed decisions about when and how to update your models.
• Prompt Optimization: Focus on iterative error correction and using code interpreters to refine your model’s capabilities continuously. This helps in addressing specific issues and improving the overall performance of your models.
• Regular Benchmarking: Continuously compare LLM performance against human benchmarks to ensure they remain competitive and effective.

Keeping your models updated and finely tuned is essential for maintaining high performance. Here are some ways companies are using successfully evaluated and iterated models

• Code Generation: Automate code tasks to boost productivity and focus on complex problem-solving.
• Error Correction: Use feedback-driven strategies for continuous debugging and optimization, making models more robust.
• Cross-Language Development: Utilize LLMs for translating code across different programming languages, improving versatilit and expanding your models to various domains.
• CI/CD Integration: Automate testing and error correction processes within your continuous integration and continuous deployment (CI/CD) pipelines. This ensures that your models are always performing at their best, even as they evolve.

Let the Experts Evaluate For You

The time to conduct LLM evaluations can range from a few days to several weeks, depending on the evaluation framework and the specific metrics and tasks being assessed. Optimization techniques such as using lower-precision grading scales and automating parts of the evaluation with LLMs can help reduce the time and cost involved.

Choosing the right model evaluation framework is essential for optimizing LLM performance. Tailor your evaluation approach to your specific needs, continuously monitor performance, and iterate on improvements to ensure your models remain accurate, efficient, and reliable.