Скачать с ютуб Optimising Open Source LLM Deployment on Cloud Run в хорошем качестве

Optimising Open Source LLM Deployment on Cloud Run

🚀 Deep Dive: Ollama vs VLLM vs HuggingFace TGI – Performance Comparison for Open-Source LLMs on Google Cloud Run I’ve just released a follow-up to my first video, “When Cloud Run Meets Deepseek”! This new instalment is a detailed performance comparison of three deployment methods for open-source LLMs: Ollama, VLLM, and HuggingFace TGI. If you’re aiming for speed, concurrency, or cost-efficiency on Google Cloud Run, here’s a closer look! 🔑 Key Insights: • Why Open-Source LLMs? Enjoy security, flexibility for fine-tuning, and cost control—excellent for enterprise scenarios. • Why Cloud Run? Take advantage of serverless scaling (from 0 to 1,000 instances!), GPU support in preview, and scale-to-zero to keep costs down. ⚙️ Performance Deep Dive: • Ollama: Straightforward to deploy and well-suited for moderate concurrency. • VLLM: Excels at concise outputs, making it ideal for shorter or mid-length responses. • HuggingFace TGI: Handles 60+ concurrent requests and 2,000+ tokens seamlessly. ✨ Distilled Models (e.g., Deepseek R1-7B): Compact, cost-effective, and surprisingly powerful for niche use cases. 💷 Cost Analysis: Combining Cloud Run with TGI can bring costs down to roughly 2.6p per user-hour at scale. 📈 Future Trends: Distilled models and innovations like NVIDIA’s Project Digits are leading to smaller, more efficient solutions with sharper performance. ⏱️ Jump to Key Sections: • 01:17 - Why Open-Source LLMs Matter • 03:04 - Why Cloud Run? • 05:06 - Ollama vs VLLM vs HuggingFace TGI • 07:26 - What’s a Distilled Model? • 10:34 - Ollama Performance • 12:36 - VLLM Performance • 15:20 - TGI Performance • 18:30 - Side-by-Side Comparison • 22:37 - Cloud Run Cost Breakdown • 23:46 - Live Demo • 37:12 - The Future of Open-Source LLMs 👉 Watch the full video for GPU utilisation stats, latency benchmarks, and live demos. If you’re exploring LLM deployments or cloud optimisation, I’d love to hear your insights! Source Code: TGI: https://github.com/richardhe-fundamen... VLLM: https://github.com/richardhe-fundamen... Ollama: https://github.com/richardhe-fundamen... #OpenSourceAI #LLM #GoogleCloud #CloudRun #AIOptimisation #TechInsights #MachineLearning #DeepSeek #DeepSeekR1