Intel Reveals Gaudi 3 AI Accelerator for Generative AI

Today marks the introduction of the Gaudi 3, a new AI accelerator that Intel says builds significantly on the performance and scalability of its predecessor, delivering 4x AI compute for BF16, a 1.5x increase in memory bandwidth, and 2x networking bandwidth for AI training and inference on popular large language models (LLMs) and multimodal models, aimed at generative AI. Each of these accelerators are powerful enough to perform 64,000 parallel operations, according to Intel.

Availability:

• OEMs: Q2 2024
• General: Q3 2024
• PCIe AIC: Q4 2024

An overview of Gaudi 3:

Key features include:

• AI-Dedicated Compute Engine
  • The Intel Gaudi 3 accelerator was purpose-built for high-performance, high-efficiency GenAI compute.  Each accelerator uniquely features a heterogenous compute engine comprised of 64 AI-custom and programmable TPCs and eight  MMEs. Each Intel Gaudi 3 MME is capable of performing an impressive 64,000 parallel operations, allowing a high degree of computational efficiency, making them adept at handling complex matrix operations, a type of computation that is fundamental to deep learning algorithms. This unique design accelerates speed and efficiency of parallel AI operations and supports multiple data types, including FP8 and BF16.  
• Memory Boost for LLM Capacity Requirements
  • 128 gigabytes (GB) of HBMe2 memory capacity, 3.7 terabytes (TB) of memory bandwidth and 96 megabytes (MB) of on-board static random access memory (SRAM) provide ample memory for processing large GenAI datasets on fewer Intel Gaudi 3s, particularly useful in serving large language and multimodal models, resulting in increased workload performance and data center cost efficiency.  
• Efficient System Scaling for Enterprise GenAI
  • Twenty-four 200 gigabit (Gb) Ethernet ports are integrated into every Intel Gaudi 3 accelerator, providing flexible and open-standard networking. They enable efficient scaling to support large compute clusters and eliminate vendor lock-in from proprietary networking fabrics. The Intel Gaudi 3 accelerator is designed to scale up and scale out efficiently from a single node to thousands to meet the expansive requirements of GenAI models. 
• Open Industry Software for Developer Productivity
  • Intel Gaudi software integrates the PyTorch framework and provides optimized Hugging Face community-based models – the most-common AI framework for GenAI developers today. This allows GenAI developers to operate at a high abstraction level for ease of use and productivity and ease of model porting across hardware types. 
• Gaudi 3 PCIe
  • New to the product line is the Gaudi 3 peripheral component interconnect express (PCIe) add-in card. Tailored to bring high efficiency with lower power, this new form factor is ideal for workloads such as fine-tuning, inference and retrieval-augmented generation (RAG). It is equipped as a full-height form factor at 600 watts, with a memory capacity of 128GB and a bandwidth of 3.7TB per second.  

Performance claims from Intel:

• 50% faster time-to-train across Llama2 7B and 13B parameters, and GPT-3 175B parameter models.  
• 50% faster inference throughput and 40% greater inference power-efficiency across Llama 7B and 70B parameters, and Falcon 180B parameter models. An even greater inference performance advantage on longer input and output sequences.
• 30% faster inferencing on Llama 7B and 70B parameters, and Falcon 180B parameter models against Nvidia H200.  

Intel on what’s coming next:

Intel Gaudi 3 accelerators’ momentum will be foundational for Falcon Shores, Intel’s next-generation graphics processing unit (GPU) for AI and high-performance computing (HPC). Falcon Shores will integrate the Intel Gaudi and Intel Xe intellectual property (IP) with a single GPU programming interface built on the Intel oneAPI specification.

Source

Join the discussion in our forums...