Categories

Tag

inferencia

View Cards
Mode Infinite
Per page Per batch 24
Clear filters
Artificial Intelligence

ONNX Runtime at the Edge: Portable, Fast Inference

A model trained in PyTorch or TensorFlow, running the same way on a server, a phone, a browser tab, or an ARM gateway on the factory floor: that is what ONNX Runtime solves. It turns the ONNX format into a genuinely portable artifact, exported once, at the cost of some peak performance versus a platform-native runtime.

3 min 329
Artificial Intelligence

Hugging Face TGI: Serving Open Models at Scale

Text Generation Inference (TGI) is the Hugging Face stack for serving open LLMs in production: continuous batching, 4-bit and 8-bit quantization, streaming, and an OpenAI-compatible API. After a brief restrictive-licence episode in 2023, it returned to Apache 2.0 in version 2.0.

4 min 281 4.4