Hugging Face's logo

openai
/
privacy-filter

Token Classification
Transformers
ONNX
Safetensors
Transformers.js
openai_privacy_filter
Model card Files
xet
 Community
29

vllm or something

#25
by prudant - opened
Discussion
prudant

nice work, can be served for production environments with vllm or similars? regards
Entro-AI
•
edited Jun 28

https://github.com/vllm-project/vllm/pull/41026/changes#diff-c24f24c74950d260bad75de22c22722967d356be0ed2b37b10d81064478744e2
andynoodles

Tried with vllm/vllm-openai:cu129-nightly
Currently does not work for 5090. RTX Pro 6000, GB10
Require Flash Attention 3 to work (H100, H200...)
Entro-AI

Tried with vllm/vllm-openai:cu129-nightly
Currently does not work for 5090. RTX Pro 6000, GB10
Require Flash Attention 3 to work (H100, H200...)

Try with TRITON BACKEND instead of FA3

andynoodles
•
edited 24 days ago

You would know it won't work if you tried, for this model, vllm's implementation requires you to use FA3.
dos2
•
edited 9 days ago

Using Triton Backend worked for me with v0.25.1 on a L40s GPU

Sign up or log in to comment