Accelerating on-device AI: A look at Arm and Google AI Edge optimization

Arm SME2 and Google AI Edge enable efficient on-device AI inference by combining hardware acceleration with optimized software tools.

• •Arm Scalable Matrix Extension 2 (SME2) integrates matrix-compute units into CPU, delivering up to 5x faster inference for generative AI workloads
• •Google AI Edge provides LiteRT, AI Edge Quantizer, and Model Explorer for streamlined model optimization and deployment
• •LiteRT-Torch enables direct conversion of PyTorch models to .tflite format with automatic quantization support
• •Model Explorer visualizes compute-intensive operators to identify quantization-safe layers for INT8 optimization
• •Demonstrates 2x inference speedup and 4x memory reduction for Stable Audio Open Small model on Arm CPUs and M4 Macs