Server is at capacity
• 7
ML intern explorers
Team
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
Bc-AI
updated 2
Spaces 1 day ago
Bc-AI
published a
Space 2 days ago
Plans
2
#1 opened 4 days ago
by 
Reality123b
Plans
2
#1 opened 4 days ago
by Reality123b
Are you sure there's enough credits?
28
#1 opened 10 days ago
by 
vovaRL
Are you sure there's enough credits?
28
#1 opened 10 days ago
by vovaRL
Bc-AI
updated a
Space 6 days ago
Bc-AI
published a
Space 6 days ago
Bc-AI
updated 3
Spaces 7 days ago
Bc-AI
published a
Space 7 days ago
Bc-AI
updated a
dataset 9 days ago
cmpatino
updated a
bucket 9 days ago
Bc-AI
published a
Space 9 days ago
Frostbittn
updated a
collection 10 days ago
Are you sure there's enough credits?
28
#1 opened 10 days ago
by vovaRL
alvarobartt
posted an update 3 months ago
Post
573
Open agents on AWS SageMaker AI with open models from the Hugging Face Hub!
> Deploy an open model from the Hugging Face Hub on SageMaker AI
> Connect the deployed model to Strands Agents
> Add built-in and custom tools for tool calling
> Expose external capabilities through MCP integration
> Bonus: talk to your agent and visualize traces with Gradio
https://alvarobartt.com/agents-on-aws-sagemaker
> Deploy an open model from the Hugging Face Hub on SageMaker AI
> Connect the deployed model to Strands Agents
> Add built-in and custom tools for tool calling
> Expose external capabilities through MCP integration
> Bonus: talk to your agent and visualize traces with Gradio
https://alvarobartt.com/agents-on-aws-sagemaker
alvarobartt
posted an update 3 months ago
Post
3394
Latest
TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
🧠
🏗️ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
⚡ Active params isn't the same as memory footprint, especially for sparse architectures
📦 Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
📚 KV cache can still dominate depending on context length, batch size, and concurrency
🔀 Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
🚀 Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem
hf-mem release added a breakdown of Mixture-of-Experts (MoE) memory usage!TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
🧠
hf-mem now splits MoE memory into base model weights, routed experts, and KV cache🏗️ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
⚡ Active params isn't the same as memory footprint, especially for sparse architectures
📦 Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
📚 KV cache can still dominate depending on context length, batch size, and concurrency
🔀 Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
🚀 Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem