Скачать с ютуб BitNet b1.58 2B4T Technical Report (Apr 2025) в хорошем качестве

BitNet b1.58 2B4T Technical Report (Apr 2025)

Title: BitNet b1.58 2B4T Technical Report (Apr 2025) Link: http://arxiv.org/abs/2504.12285v1 Date: April 2025 Summary: Introduces BitNet b1.58 2B4T, the first open-source 1-bit LLM at 2B scale, trained on 4T tokens. It achieves performance on par with similar sized full-precision LLMs with significant gains in computational efficiency. Model weights and inference code are released. Key Topics: 1-bit Large Language Models BitNet Model Quantization Computational Efficiency Low-bit LLMs Inference Optimization Chapters: 00:00 - Intro to BitNet 00:05 - Efficiency Explained 00:13 - Staggering Headline 00:25 - Native One-Bit Training 00:35 - Deploying AI 00:45 - Unpacking the Architecture 00:53 - Computational Cost 01:03 - BitNet as a Solution 01:16 - Four Trillion Tokens 01:26 - The Core Challenge 01:37 - One-Bit LLMs 01:50 - Binary or Turnary Values 02:04 - Native Aspect of BitNet 02:16 - Post-Training Quantization 02:32 - Performance Hit 02:44 - Bit Linear Layers 02:57 - Model Capabilities 03:08 - Absmein Scheme 03:19 - Massive Reduction 03:32 - Incredible Compression 03:41 - Eight Bit Integers 03:50 - Smart Trade Off 03:55 - Sublayer Normalization 04:07 - Squared ReLU 04:29 - Rotary Position Embeddings 04:39 - Parameter Count 04:48 - Tokenizer 05:04 - Training the Beast 05:14 - Supervised Fine-Tuning 05:24 - Learning Rate Schedule 05:36 - More Stable Training 05:52 - Cool Down Phase 06:02 - Weight Decay 06:19 - Pre-training Data 06:36 - Two-Stage Data Strategy 06:49 - SFT Data 07:09 - Chat Template 07:23 - Loss Calculation 07:38 - SFT Learning Rate 07:50 - Direct Preference Optimization 08:04 - Reward Model 08:16 - Preference Data for DPO 08:30 - DPO Training 08:41 - Model's Core Abilities 08:56 - Performance Benchmarks 09:19 - Thorough Evaluation 09:35 - Efficiency Metrics 09:49 - Energy Use 10:00 - Performance 10:17 - Overall Strong Performance 10:28 - Resource Cost 10:38 - INT4 Quantization 10:53 - Post-Training Quantization 11:08 - One-Bit Models 11:24 - Native Training Approach 11:36 - Inference 11:48 - Custom CUDA Kernels 11:58 - Pack Store Load 12:17 - On-ships RAM 12:30 - Minimizing Data Movement 12:40 - Current GPU's 12:53 - CPU Inference 13:16 - Accessibility 13:29 - Big Step for Efficient LLMs 13:44 - Open Sourcing 13:54 - Future Directions 14:06 - Final Thought 14:24 - Thanks