Leaderboard.

NVIDIA AI provides a comprehensive suite of hardware and software solutions for accelerating AI development and deployment. Their offerings include GPUs optimized for deep learning, AI software development kits (SDKs), and pre-trained AI models to enable faster innovation across various industries.

The Hugging Face Transformers training script simplifies the process of training and fine-tuning transformer models for various NLP tasks. It provides a high-level API and pre-built training loops, enabling users to quickly adapt pre-trained models to their specific datasets and objectives.

A Python script using PyTorch for training and evaluating image classification models. It provides a modular structure for defining datasets, models, training loops, and evaluation metrics, enabling researchers and practitioners to quickly prototype and deploy image classification solutions.

MLPerf Training is a suite of benchmarks that measure the time it takes to train various machine learning models on different hardware and software platforms. It provides a standardized way to compare the performance of different AI training systems, driving innovation in hardware and software optimization for AI workloads.

The TensorFlow Model Garden is a repository containing a collection of example implementations for state-of-the-art (SOTA) machine learning models and modeling solutions for TensorFlow. It provides a wide variety of models, pre-trained weights, and scripts to help users quickly prototype and deploy TensorFlow-based AI solutions.

HELM is a living benchmark designed to provide a comprehensive and holistic evaluation of language models across a wide range of scenarios and metrics. It aims to move beyond single-number evaluations by assessing models on factors like truthfulness, calibration, fairness, robustness, and efficiency, providing a more nuanced understanding of their capabilities and limitations.

Amazon SageMaker is a fully managed machine learning service that enables data scientists and developers to build, train, and deploy machine learning models quickly. It provides a suite of tools and services covering the entire ML lifecycle, from data preparation to model deployment and monitoring.

The TensorFlow Model Optimization Toolkit script provides tools and techniques to optimize TensorFlow models for deployment, including quantization, pruning, and clustering. It reduces model size and improves inference speed, making models more suitable for edge devices and resource-constrained environments.

A Python script leveraging scikit-learn to comprehensively evaluate machine learning models. It calculates various performance metrics (e.g., accuracy, precision, recall, F1-score, AUC) and generates visualizations (e.g., confusion matrices, ROC curves) to provide insights into model behavior and facilitate informed decision-making.

LCEL is a declarative way to compose chains of language models and other primitives in LangChain. This script demonstrates how to use LCEL to build complex AI pipelines with features like streaming, parallel execution, and retry mechanisms, enabling developers to create robust and scalable AI applications.

Detects anomalies in live system telemetry, runs deterministic diagnostics from the organization's top remediation runbooks, and autonomously resolves up to 40% of standard incidents without human intervention. Operates within strict change-window and read-only access constraints, with mandatory human-in-the-loop approval for any remediation touching production data or falling outside predefined runbooks. Reduces mean-time-to-recovery and augments on-call teams.

The AI2 Reasoning Challenge (ARC) is a question-answering dataset designed to encourage research in advanced question-answering. It consists of grade-school science questions specifically crafted to require reasoning beyond simple fact retrieval, posing a significant challenge for AI models.

The canonical large-scale visual recognition benchmark containing 1.28 million training images across 1,000 object categories. ImageNet-1K underpins the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and has driven the majority of deep learning breakthroughs in computer vision since 2012.

This script provides a streamlined method for performing image generation using Stable Diffusion XL Turbo. It leverages optimized inference techniques to achieve faster generation speeds, making it suitable for real-time applications and interactive experiences.

The Databricks Feature Store provides a centralized repository for managing and sharing machine learning features. Its integration with MLflow enables seamless tracking of feature usage in ML models, ensuring reproducibility and simplifying model deployment workflows by automatically packaging feature dependencies.

Introduced BERT, a bidirectional Transformer pre-trained on masked language modeling and next sentence prediction. Established the pretrain-then-fine-tune paradigm that dominated NLP for years and achieved state-of-the-art on 11 NLP benchmarks.

Continuously observes CI/CD pipelines, code repositories, and incident logs. Detects deployment anomalies the moment thresholds breach, safely rolls back anomalous releases using historical context, and triggers automated fixes — all without waiting for a human on-call engineer. Operates within strict rollback policies including blast-radius limits and change-window enforcement to prevent cascading failures.

Microsoft COCO (Common Objects in Context) 2017 provides 118K training images with 860K object instances annotated with bounding boxes, segmentation masks, keypoints, and captions across 80 object categories. It remains the primary benchmark for object detection and instance segmentation research.

Databricks is a unified data analytics platform built on Apache Spark, providing tools for data engineering, data science, and machine learning. It enables organizations to process large datasets, build and deploy ML models, and collaborate across teams.

Introduced CLIP (Contrastive Language-Image Pre-training), a model trained on 400 million image-text pairs using contrastive learning that achieves remarkable zero-shot transfer to diverse vision tasks. CLIP became foundational for vision-language alignment and generative AI pipelines.

Introduced chain-of-thought prompting, a simple technique of providing exemplars with step-by-step reasoning traces in few-shot prompts. This approach dramatically improves LLM performance on arithmetic, commonsense, and symbolic reasoning tasks, with the effect emerging at approximately 100B parameters.

Introduced Latent Diffusion Models (LDMs), which perform the diffusion process in a compressed latent space rather than pixel space, dramatically reducing computational cost while maintaining image quality. This work underpins Stable Diffusion, the most widely used open-source image generation model.

Introduced GPT-3, a 175B parameter language model demonstrating remarkable few-shot learning capabilities across diverse tasks. Showed that scaling model size dramatically improves in-context learning without gradient updates, reshaping the field.

The RCSB Protein Data Bank (PDB) is the single worldwide archive of experimentally determined 3D structures of proteins, nucleic acids, and complex assemblies, currently containing over 220,000 biological macromolecular structures determined by X-ray crystallography, NMR, and cryo-EM. It is the foundational structural dataset for computational biology and was used to train and validate AlphaFold2 and other structure-prediction models.

Introduced the Vision Transformer (ViT), demonstrating that a pure transformer applied directly to sequences of image patches achieves state-of-the-art performance on image classification when pretrained on large datasets. The paper challenged the dominance of convolutional neural networks in computer vision.

PyTorch Geometric (PyG) is a library built upon PyTorch to facilitate the development of graph neural networks (GNNs). It provides data handling utilities, learning methods on graphs and other irregular structures, and benchmark datasets for various graph-related tasks.

Presents InstructGPT, which uses Reinforcement Learning from Human Feedback (RLHF) to align GPT-3 with human intent. By fine-tuning on human demonstrations and training a reward model on human preference comparisons, InstructGPT produces outputs that human evaluators prefer to GPT-3 outputs despite having 100× fewer parameters.

ImageNet (ILSVRC) is the foundational large-scale visual recognition benchmark with 1.2 million training images across 1,000 object categories. Top-1 and Top-5 accuracy on the validation set have been the standard measure of progress in image classification for over a decade.

Introduces Retrieval-Augmented Generation (RAG), combining parametric memory (language model weights) with non-parametric memory (dense retrieval over Wikipedia) for knowledge-intensive NLP tasks. RAG models achieve state-of-the-art on open-domain QA benchmarks and produce more specific, factual, and diverse responses than pure parametric models.

PPO introduces a clipped surrogate objective that constrains policy update step sizes, achieving the stability of trust-region methods (TRPO) with the simplicity and scalability of first-order optimizers. It quickly became the dominant RL algorithm for training large language models with human feedback.

AlphaFold 2 achieves atomic-level accuracy in protein structure prediction by combining evolutionary information from multiple sequence alignments with a novel Evoformer architecture and structure module, solving a 50-year grand challenge in biology. Its predictions have been released for virtually all known proteins and have accelerated drug discovery, enzyme design, and structural biology worldwide.

RoboSuite is a simulation framework and benchmark suite for robot learning. It provides a standardized set of environments and tasks for training and evaluating reinforcement learning algorithms in robotics, focusing on manipulation and locomotion tasks with realistic physics and sensor models.

UniProt (Universal Protein Resource) is the world's comprehensive, freely accessible protein sequence and functional information database, maintained by a consortium of EMBL-EBI, SIB, and PIR. It contains over 250 million protein sequences in UniParc, with 570,000+ manually reviewed entries in SwissProt providing expert-curated functional annotations, and serves as the gold-standard training source for protein language models.

Massive Multitask Language Understanding (MMLU) is a benchmark covering 57 academic subjects from STEM to humanities, with 14,000+ multiple-choice questions at undergraduate and professional level. It has become the de facto standard for measuring broad world knowledge and academic reasoning in LLMs.

AssemblyAI provides a Speech-to-Text API that allows developers to transcribe audio and video files with high accuracy. Their platform offers features like speaker diarization, sentiment analysis, and content moderation, making it a comprehensive solution for audio intelligence.

The AI2 Reasoning Challenge (ARC) is a question-answering dataset designed to evaluate advanced reasoning capabilities in AI systems. It consists of elementary-level science questions specifically crafted to be difficult for retrieval-based methods and require deeper understanding and reasoning to answer correctly.

COCO Detection is the standard benchmark for object detection and instance segmentation, featuring 330,000 images with over 1.5 million annotated instances across 80 object categories. Mean Average Precision (mAP) at various IoU thresholds is the primary metric.

The processed Wikipedia dataset is a cleaned and tokenized version of Wikipedia dumps covering 20+ languages, available via Hugging Face Datasets. With HTML stripped and paragraph structure preserved, it is one of the most universally used pretraining corpora and a standard knowledge-grounding source for retrieval-augmented generation (RAG) baselines and open-domain QA systems.

The full text dump of Wikipedia articles available in over 300 languages, regularly updated and distributed by the Wikimedia Foundation. It is one of the most universally included components in language model pretraining pipelines due to its high factual density, editorial quality, and broad topical coverage.

LibriSpeech is a corpus of approximately 1,000 hours of 16kHz read English speech derived from LibriVox audiobooks, split into clean and other subsets of 100h and 360h for training, with dedicated development and test sets. It has become the de facto standard benchmark for English ASR systems.

Grade School Math 8K is a dataset of 8,500 high-quality linguistically diverse grade school math word problems requiring 2-8 step reasoning. Created by OpenAI, GSM8K is widely used for evaluating multi-step arithmetic reasoning and the effectiveness of chain-of-thought prompting.

TensorFlow Quantum (TFQ) is a library for building quantum machine learning models. It allows researchers to construct and train hybrid quantum-classical models by leveraging TensorFlow's infrastructure for classical computation and quantum simulators or quantum hardware for quantum computation.

LibriSpeech is the standard English automatic speech recognition (ASR) benchmark derived from LibriVox audiobooks, containing 1,000 hours of read speech at 16kHz. Word Error Rate (WER) on clean and noisy test splits drives competitive progress in ASR research.

The largest platform for sharing and deploying machine learning models, datasets, and applications. Provides the Transformers library, Inference API, Spaces for demos, and a vibrant open-source AI community.

OpenAI's frontier model with advanced reasoning, native multimodal understanding, and robust function calling. Designed for complex enterprise workflows and agentic applications.

Native integration between LangChain and OpenAI's GPT models. Provides seamless access to chat completions, embeddings, and function calling through LangChain's unified interface. Supports streaming, tool use, and structured output via the langchain-openai package.

Hugging Face is the GitHub of AI, providing the world's largest open model hub, dataset repository, and ML collaboration platform. Its Transformers library is the de-facto standard for working with open-weight models, and the Hugging Face Hub hosts hundreds of thousands of models and datasets. Its Spaces platform allows AI demos to be deployed instantly.

Leverages knowledge from a source domain to improve model performance on a target domain with limited labeled data. A foundational technique for reducing training costs and accelerating model development across diverse applications.

OpenAI's natively multimodal flagship model processing text, image, and audio inputs with a single unified architecture. Delivers GPT-4 Turbo-level intelligence at 2x speed and 50% lower cost, with breakthrough real-time voice capabilities.

Anthropic's most capable model featuring advanced reasoning, coding, and multimodal capabilities. Excels at complex analysis, agentic tasks, and extended thinking with industry-leading safety.

OpenAI's breakthrough large language model that demonstrated a significant leap in reasoning and factual accuracy over GPT-3.5. Widely adopted across enterprise and developer workflows for code generation, analysis, and complex problem-solving.

Anthropic's breakout model that surpassed Claude 3 Opus at Sonnet-tier pricing, setting new industry benchmarks for coding. Introduced computer use capability and became the most popular model on the API due to its exceptional intelligence-to-cost ratio.

Midjourney V6 represents a major leap in photorealism, prompt adherence, and artistic coherence, setting a new industry benchmark for AI image generation quality. It introduced native text rendering within images and dramatically improved its understanding of complex, multi-subject prompts.

The MLflow integration with Databricks provides a managed MLflow service within the Databricks platform. It simplifies the process of tracking experiments, managing models, and deploying them to production by leveraging Databricks' scalable infrastructure and collaborative environment.

OpenAI's state-of-the-art open-source automatic speech recognition model trained on 680K hours of multilingual audio. Supports 99 languages with near-human accuracy and includes translation, timestamp, and language detection capabilities.

Maps the entire dependency tree across an organization's codebases, tests library updates in isolated sandbox environments, writes localized unit tests to verify compatibility, and submits fully validated pull requests that respect architectural constraints. Prevents the cascade-of-breaking-changes problem that plagues manual dependency updates, where an LLM taking a prompt literally would introduce version conflicts or accidentally remove necessary features.

Guides LLMs to produce step-by-step reasoning before arriving at a final answer. Dramatically improves performance on math, logic, and multi-step problems by making the model's reasoning process explicit and verifiable.

The foundational discipline of crafting effective prompts to elicit desired behaviors from language models. Covers system prompt design, instruction formatting, output structuring, temperature tuning, and iterative prompt refinement techniques.

GitHub Copilot integrates into VS Code as a first-party extension, delivering inline ghost-text completions, multi-line suggestions, and a dedicated Copilot Chat panel for conversational refactoring, test generation, and documentation. It leverages Codex and GPT-4 models under the hood, with workspace-aware context from open tabs and the current file.

BERT (Bidirectional Encoder Representations from Transformers) is Google's landmark 2018 language model that introduced the bidirectional pre-training paradigm using masked language modeling and next sentence prediction. It revolutionized NLP by demonstrating that a single pre-trained model could achieve state-of-the-art results across dozens of downstream tasks with minimal fine-tuning.

Google DeepMind's flagship thinking model with native multimodal understanding across text, images, audio, and video. Excels at complex reasoning, code generation, and agentic tasks with a million-token context window.

ADE20K is the benchmark for semantic scene parsing, containing 25,000 images densely annotated with 150 semantic categories. Mean Intersection over Union (mIoU) is the standard metric, and it drives progress in perception systems for autonomous driving, robotics, and scene understanding.

Official Meta Llama model weights distributed through the HuggingFace Hub under Meta's community license. Covers Llama 3.1, 3.2, and 3.3 variants from 1B to 405B parameters with full transformers, TGI, and vLLM compatibility. HuggingFace serves as the primary public distribution channel for Meta's open-weight releases.

Grade School Math 8K benchmark with 8,500 linguistically diverse grade school math word problems requiring 2-8 step reasoning. Tests basic mathematical reasoning and arithmetic with problems that require sequential multi-step solutions.

Amazon Web Services is the world's largest cloud provider and offers the most comprehensive set of AI and machine learning services, including Amazon Bedrock for managed foundation model APIs, SageMaker for MLOps, Rekognition for computer vision, and Alexa for voice AI. AWS Bedrock gives enterprises access to models from Anthropic, Meta, Mistral, Cohere, and others through a unified API.

Generates functional code from natural language descriptions, specifications, or partial implementations. Covers multiple languages and frameworks with support for boilerplate scaffolding, algorithm implementation, and API integration patterns.

TensorFlow Lite is Google's lightweight ML framework designed for on-device inference on mobile, embedded, and IoT devices. It enables deploying trained models with minimal latency and no network dependency, supporting a wide range of hardware accelerators including GPU, DSP, and NPU.

Battle-tested workflow scheduler for authoring, scheduling, and monitoring data and ML pipelines as directed acyclic graphs. The ML ecosystem around Airflow includes providers for SageMaker, Vertex AI, MLflow, and all major cloud AI services.

OpenAI's managed agent platform for building custom AI assistants with persistent threads, built-in code interpreter, file search, and function calling. Handles conversation state, tool orchestration, and context management so developers can focus on business logic.

Microsoft's autonomous agent within the Copilot ecosystem that operates across Microsoft 365 apps to automate business processes. Handles email triage, meeting preparation, document summarization, and cross-app workflow automation with enterprise-grade security.

Human-validated subset of SWE-bench containing 500 problems verified by software engineers for correctness, clarity, and solvability. Provides a more reliable signal than the full SWE-bench by filtering out ambiguous or under-specified issues.

Stability AI's high-resolution image generation model producing photorealistic and artistic images at 1024x1024 resolution. Features a two-stage architecture with a base model and refiner for enhanced detail and compositional quality.

LangChain Inc is the company behind the most widely adopted LLM orchestration framework in the AI ecosystem. LangChain provides composable abstractions for building LLM-powered applications, while its LangSmith platform offers observability and evaluation tooling, and LangGraph enables the construction of stateful, multi-actor agent workflows.

Microsoft Azure AI is the AI services division of Microsoft's cloud platform, uniquely positioned as the exclusive cloud partner of OpenAI. Through Azure OpenAI Service, enterprises access GPT-4, DALL-E, and Whisper with enterprise-grade compliance and data residency guarantees. Microsoft has deeply integrated AI across its product suite including Copilot for Microsoft 365, GitHub Copilot, and Azure AI Foundry.

The foundational JavaScript 3D library for rendering GPU-accelerated graphics in the browser via WebGL, with a growing ecosystem of AI-generated geometry, procedural shaders, and LLM-driven scene graph manipulation. Three.js powers the majority of web-based spatial AI visualizations.

The AMD Instinct MI350X is a data center GPU designed for high-performance computing and AI workloads. It utilizes a CDNA 4 architecture and features HBM3E memory, offering substantial improvements in memory bandwidth and capacity compared to previous generations, making it suitable for large language model training and inference.

Enables LLMs to invoke external functions by generating structured JSON arguments matching defined schemas. Supports parallel function calls, error handling, and chained invocations for complex multi-step tool interactions.

An adaptive tutoring agent that dynamically adjusts difficulty, pacing, and instructional modality based on individual learner performance signals. It maintains a persistent knowledge model per student, identifies misconceptions through Socratic questioning, and routes learners to mastery via spaced-repetition scheduling.

The analytics engineering framework that transforms raw warehouse data into clean, tested, and documented datasets ready for ML and AI. dbt's model graph, column-level lineage, and semantic layer make it the backbone of production feature engineering pipelines.

Predicts user preferences by identifying patterns from collective user-item interaction histories, using memory-based neighborhood methods or model-based matrix factorization and neural approaches. The backbone of recommendation systems at scale across e-commerce, streaming, and social platforms.

Teaches LLMs to perform tasks by providing a small number of input-output examples in the prompt. Enables rapid task adaptation without fine-tuning by demonstrating the desired pattern through carefully selected, representative examples.

Official LangChain integration for Anthropic's Claude model family. Exposes Claude's extended context window, vision capabilities, and tool use through LangChain's standard chat model interface. Supports streaming and the full Messages API via the langchain-anthropic package.

Direct integration pairing Pinecone's managed vector database with OpenAI's text-embedding-3 models. Commonly used pattern for production RAG systems where OpenAI generates dense vectors and Pinecone handles ANN retrieval at scale. Supports serverless and pod-based indexes with metadata filtering.

Apache Spark's built-in machine learning library for distributed, large-scale ML on data lakes and warehouses. MLlib provides scalable algorithms for classification, regression, clustering, and collaborative filtering, plus a pipeline API for feature engineering.

NVIDIA's flagship consumer GPU based on Ada Lovelace. Has become popular for local LLM inference and fine-tuning due to its 24GB GDDR6X memory and high performance-per-dollar ratio, enabling on-premise AI workloads without data center costs.

Weights & Biases integrates directly into Hugging Face Trainer and PEFT via a built-in report_to callback, logging training loss curves, GPU utilization, gradient norms, and hyperparameters to shareable W&B runs. The integration supports sweep-based hyperparameter optimization and artifact versioning for model checkpoints.

Maps structural dependencies, architectural patterns, and historical technical decisions across enterprise codebases. When a critical service fails and the original developers are unavailable, this agent produces a semantic architecture map — dependency graphs, hotspot analysis, and knowledge gap identification — in minutes instead of weeks. Integrates deeply with repositories to understand code as architecture, not just text.

A fully-autonomous customer support agent that unifies conversations across chat, email, SMS, and social DMs into a single threaded context window. It resolves tier-1 and tier-2 tickets using a retrieval-augmented knowledge base and maintains CSAT targets through sentiment-aware tone calibration.

Microsoft's multi-agent conversation framework enabling multiple LLM agents to converse, collaborate, and solve tasks through automated chat. Supports customizable agent behaviors, human-in-the-loop, and code execution sandboxing.

The AMD Instinct MI400A is a data center accelerator designed for high-performance computing and AI workloads. It integrates CPU and GPU cores on a single chip, aiming to improve performance and efficiency for demanding AI applications.

The Stability AI Platform provides API access to Stability AI's suite of generative image, video, and audio models including Stable Diffusion 3.5 and Stable Video Diffusion, enabling developers to build creative AI applications at scale. It offers both hosted API endpoints and open-weight models for on-premises deployment.

MediaPipe is Google's cross-platform framework for building perception pipelines that run on-device in real time. It provides production-ready solutions for tasks like hand tracking, face detection, pose estimation, and object detection across Android, iOS, web, and desktop.

Equips AI agents with the ability to select and use appropriate tools from a defined toolkit to accomplish tasks. Covers tool selection logic, input marshalling, output interpretation, and fallback strategies when tools fail or return unexpected results.

AI-powered answer engine that combines real-time web search with LLM synthesis to provide cited, accurate answers. Features multi-step research capabilities, source verification, and conversational follow-up for deep topic exploration.

Teaches integration and optimization of automatic speech recognition (ASR) systems — from Whisper to streaming cloud APIs — for agentic voice pipelines. Covers language identification, word error rate reduction, punctuation restoration, and handling noisy audio environments.

The Cerebras WSE-4 is the fourth generation wafer-scale processor designed specifically for AI compute. It features a massive array of compute cores fabricated on a single silicon wafer, enabling extremely high bandwidth and low latency for large AI models.

The AMD Instinct MI400 series is a family of data center GPUs designed for high-performance computing and AI workloads. It leverages AMD's CDNA 4 architecture and offers significant improvements in performance and energy efficiency compared to previous generations, targeting large-scale AI training and inference.

Python-first framework for building interactive data applications and ML demos in minutes with no frontend experience required. Streamlit's reactive execution model, built-in widgets, and LLM streaming components make it the go-to tool for AI prototype UIs.

Google Cloud AI provides enterprise access to Google DeepMind's Gemini models and a comprehensive suite of managed AI services via Vertex AI. As the creator of the Transformer architecture and TensorFlow, Google Cloud offers unmatched AI infrastructure including custom TPUs, a full MLOps platform, and pre-built APIs for vision, speech, and natural language processing.

Production-grade ASR pipeline using OpenAI Whisper or faster-whisper with VAD-based chunking, speaker timestamp alignment, and SRT/VTT subtitle export. Handles long-form audio via sliding window segmentation and automatic language detection.