Shaping capabilities with token-level data filtering
Paper
β’
2601.21571
β’
Published
β’
25
AI & ML interests
None defined yet.
Recent Activity
View all activity
julien-cΒ
submitted
a
paper
to Daily Papers
10 days ago
Post
489
Interesting article: use Claude Code to help open models write CUDA kernels (for eg) by turning CC traces into Skills. They made a library out of it π
https://huggingface.co/blog/upskill
https://huggingface.co/blog/upskill
nielsrΒ
submitted
a
paper
to Daily Papers
11 days ago
IlyasMoutawwakilΒ
posted
an
update
13 days ago
Post
2969
Transformers v5 just landed! π
It significantly unifies and reduces modeling code across architectures, while opening the door to a whole new class of performance optimizations.
My favorite new feature? π€
The new dynamic weight loader + converter. Hereβs why π
Over the last few months, the core Transformers maintainers built an incredibly fast weight loader, capable of converting tensors on the fly while loading them in parallel threads. This means weβre no longer constrained by how parameters are laid out inside the safetensors weight files.
In practice, this unlocks two big things:
- Much more modular modeling code. You can now clearly see how architectures build on top of each other (DeepSeek v2 β v3, Qwen v2 β v3 β MoE, etc.). This makes shared bottlenecks obvious and lets us optimize the right building blocks once, for all model families.
- Performance optimizations beyond what torch.compile can do alone. torch.compile operates on the computation graph, but it canβt change parameter layouts. With the new loader, we can restructure weights at load time: fusing MoE expert projections, merging attention QKV projections, and enabling more compute-dense kernels that simply werenβt possible before.
Personally, I'm honored to have contributed in this direction, including the work on optimizing MoE implementations and making modeling code more torch-exportable, so these optimizations can be ported cleanly across runtimes.
Overall, Transformers v5 is a strong signal of where the community and industry are converging: Modularity and Performance, without sacrificing Flexibility.
Transformers v5 makes its signature from_pretrained an entrypoint where you can mix and match:
- Parallelism
- Quantization
- Custom kernels
- Flash/Paged attention
- Continuous batching
- ...
Kudos to everyone involved! I highly recommend the:
Release notes: https://github.com/huggingface/transformers/releases/tag/v5.0.0
Blog post: https://huggingface.co/blog/transformers-v5
It significantly unifies and reduces modeling code across architectures, while opening the door to a whole new class of performance optimizations.
My favorite new feature? π€
The new dynamic weight loader + converter. Hereβs why π
Over the last few months, the core Transformers maintainers built an incredibly fast weight loader, capable of converting tensors on the fly while loading them in parallel threads. This means weβre no longer constrained by how parameters are laid out inside the safetensors weight files.
In practice, this unlocks two big things:
- Much more modular modeling code. You can now clearly see how architectures build on top of each other (DeepSeek v2 β v3, Qwen v2 β v3 β MoE, etc.). This makes shared bottlenecks obvious and lets us optimize the right building blocks once, for all model families.
- Performance optimizations beyond what torch.compile can do alone. torch.compile operates on the computation graph, but it canβt change parameter layouts. With the new loader, we can restructure weights at load time: fusing MoE expert projections, merging attention QKV projections, and enabling more compute-dense kernels that simply werenβt possible before.
Personally, I'm honored to have contributed in this direction, including the work on optimizing MoE implementations and making modeling code more torch-exportable, so these optimizations can be ported cleanly across runtimes.
Overall, Transformers v5 is a strong signal of where the community and industry are converging: Modularity and Performance, without sacrificing Flexibility.
Transformers v5 makes its signature from_pretrained an entrypoint where you can mix and match:
- Parallelism
- Quantization
- Custom kernels
- Flash/Paged attention
- Continuous batching
- ...
Kudos to everyone involved! I highly recommend the:
Release notes: https://github.com/huggingface/transformers/releases/tag/v5.0.0
Blog post: https://huggingface.co/blog/transformers-v5
-
- 3 replies
IlyasMoutawwakilΒ
posted
an
update
18 days ago
Post
2342
After 2 months of refinement, I'm happy to announce that a lot of Transformers' modeling code is now significantly more torch-compile & export-friendly π₯
Why it had to be done π
PyTorch's Dynamo compiler is increasingly becoming the default interoperability layer for ML systems. Anything that relies on torch.export or torch.compile, from model optimization to cross-framework integrations, benefits directly when models can be captured as a single dynamo-traced graph !
Transformers models are now easier to:
βοΈ Compile end-to-end with torch.compile backends
π¦ Export reliably via torch.export and torch.onnx.export
π Deploy to ONNX / ONNX Runtime, Intel Corporation's OpenVINO, NVIDIA AutoDeploy (TRT-LLM), AMD's Quark, Meta's Executorch and more hardware-specific runtimes.
This work aims at unblocking entire TorchDynamo-based toolchains that rely on exporting Transformers across runtimes and accelerators.
We are doubling down on Transformers commitment to be a first-class citizen of the PyTorch ecosystem, more exportable, more optimizable, and easier to deploy everywhere.
There are definitely some edge-cases that we still haven't addressed so don't hesitate to try compiling / exporting your favorite transformers and to open issues / PRs.
PR in the comments ! More updates coming coming soon !
Why it had to be done π
PyTorch's Dynamo compiler is increasingly becoming the default interoperability layer for ML systems. Anything that relies on torch.export or torch.compile, from model optimization to cross-framework integrations, benefits directly when models can be captured as a single dynamo-traced graph !
Transformers models are now easier to:
βοΈ Compile end-to-end with torch.compile backends
π¦ Export reliably via torch.export and torch.onnx.export
π Deploy to ONNX / ONNX Runtime, Intel Corporation's OpenVINO, NVIDIA AutoDeploy (TRT-LLM), AMD's Quark, Meta's Executorch and more hardware-specific runtimes.
This work aims at unblocking entire TorchDynamo-based toolchains that rely on exporting Transformers across runtimes and accelerators.
We are doubling down on Transformers commitment to be a first-class citizen of the PyTorch ecosystem, more exportable, more optimizable, and easier to deploy everywhere.
There are definitely some edge-cases that we still haven't addressed so don't hesitate to try compiling / exporting your favorite transformers and to open issues / PRs.
PR in the comments ! More updates coming coming soon !
-
- 1 reply
nielsrΒ
submitted
a
paper
to Daily Papers
28 days ago
nielsrΒ
submitted
a
paper
to Daily Papers
about 2 months ago
Post
3395
Nvidia is on a roll lately. Nemotron 3 Nano is my new fav local model, but here's the real flex: they published the entire evaluation setup. Configs, prompts, logs, all of it. This is how you do open models π₯
https://huggingface.co/blog/nvidia/nemotron-3-nano-evaluation-recipe
https://huggingface.co/blog/nvidia/nemotron-3-nano-evaluation-recipe
Post
2767
Muon vs MuonClip vs Muon+Adamw
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fineβtuning? We ran headβtoβhead tests on Qwen3β4B (10k+ highβquality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradientβnorm spikes made training unstable. MuonClip (Kimi K2βs clipping) stabilizes long pretraining runs, yet in our smallβscale fineβtune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muonβs spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fineβtuning? We ran headβtoβhead tests on Qwen3β4B (10k+ highβquality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradientβnorm spikes made training unstable. MuonClip (Kimi K2βs clipping) stabilizes long pretraining runs, yet in our smallβscale fineβtune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muonβs spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Post
2613
I tested Muon vs MuonClip vs Muon+AdamW for fine-tuning LLMs
Just published a blog on that, Read here π https://huggingface.co/blog/KingNish/optimizer-part1
Just published a blog on that, Read here π https://huggingface.co/blog/KingNish/optimizer-part1
- 1 reply
mrfakenameΒ
posted
an
update
2 months ago
Post
13589
Excited to share that I've joined the Hugging Face Fellows program! π€
Looking forward to contributing to & working more closely with the open-source ecosystem - huge thanks to everyone who's supported me on this journey! π
Looking forward to contributing to & working more closely with the open-source ecosystem - huge thanks to everyone who's supported me on this journey! π
christopherΒ
authored
a
paper
2 months ago
reach-vbΒ
authored
a
paper
2 months ago
nroggendorffΒ
posted
an
update
3 months ago
Post
2936
I am now being charged for paused and unstarted spaces out of the blue.
I think this is it, folks. o7
The unstarted spaces I can get behind. I would've appreciated a warning email first, but whatever. However, every time I restart the active usage goes up, despite all of my spaces being moved to CPU (free), and being paused.
I think this is it, folks. o7
The unstarted spaces I can get behind. I would've appreciated a warning email first, but whatever. However, every time I restart the active usage goes up, despite all of my spaces being moved to CPU (free), and being paused.
-
- 13 replies
nroggendorffΒ
posted
an
update
3 months ago
Post
2383
Developing with ZeroGPU without a PRO account is painful. They give you so many requests at once, but then have like a 24 hour cooldown. I vote less requests in a batch, but then a shorter cooldown.
or just less of a cooldown, but i understand if that is not allowed
or just less of a cooldown, but i understand if that is not allowed
-
-
- 3 replies
GeorgeBredisΒ
authored
a
paper
3 months ago
Post
1043
The #1 trending AI/ML dataset today π
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Post
715
The new King πhas arrived!
Moonshot AI now the top model on Hugging Face π₯
moonshotai/Kimi-K2-Thinking
Moonshot AI now the top model on Hugging Face π₯
moonshotai/Kimi-K2-Thinking
Post
2809
πΈπ€You donβt need 100 GPUs to train something amazing!
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on π€ :
HuggingFaceTB/smol-training-playbook
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on π€ :
HuggingFaceTB/smol-training-playbook
DmitryRyuminΒ
posted
an
update
3 months ago
Post
1302
πποΈπ New Research Alert - ICCV 2025 (Poster)! πποΈπ
π Title: Is Less More? Exploring Token Condensation as Training-Free Test-Time Adaptation π
π Description: Token Condensation as Adaptation (TCA) improves the performance and efficiency of Vision Language Models in zero-shot inference by introducing domain anchor tokens.
π₯ Authors: Zixin Wang, Dong Gong, Sen Wang, Zi Huang, Yadan Luo
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation (2410.14729)
π Repository: https://github.com/Jo-wang/TCA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Session 1: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/session-1.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TestTimeAdaptation #TokenCondensation #VisionLanguageModels #TrainingFreeAdaptation #ZeroShotLearning #EfficientAI #AI #ICCV2025 #ResearchHighlight
π Title: Is Less More? Exploring Token Condensation as Training-Free Test-Time Adaptation π
π Description: Token Condensation as Adaptation (TCA) improves the performance and efficiency of Vision Language Models in zero-shot inference by introducing domain anchor tokens.
π₯ Authors: Zixin Wang, Dong Gong, Sen Wang, Zi Huang, Yadan Luo
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation (2410.14729)
π Repository: https://github.com/Jo-wang/TCA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Session 1: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/session-1.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TestTimeAdaptation #TokenCondensation #VisionLanguageModels #TrainingFreeAdaptation #ZeroShotLearning #EfficientAI #AI #ICCV2025 #ResearchHighlight