In short
- Hugging Face and AWS released a tool showing whether popular models run unmodified on Trainium chips.
- It covers only text and image-text models, and a PASS says nothing about speed.
The short version: Hugging Face and AWS launched a public page (a Space) where you can check whether popular models from the Hugging Face Hub run unmodified on AWS Trainium chips. The post is by David Corvoysier.
What changed
- Views: “Selected” (a list picked by the authors), “Trending” and “Downloads”.
- For each model you see the result, the number of Trainium cores used, wall time including compilation, and peak device memory.
- For a model you get the packages and a
run.pyscript you start withpython3 run.py. - You can skip the initial compilation by pulling ready graphs from a public cache bucket.
- Results live in the public
aws-neuron/torch-neuronx-model-supportdataset.
What you need
A trn2 instance with the Neuron software (for example the AWS Neuron Deep Learning Container) and the torch-neuronx package. To download the cache you need the hf tool.
How to read this
The tests cover only models that fit on 4 Trainium2 devices (up to 16 cores), generate 32 tokens, and apply only to text and image-text Transformers models. According to the authors a PASS says nothing about throughput or longer outputs. Treat it as a first filter, not a guarantee. I write more about running models on your own hardware in the hub on local AI.