Paper Group: Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

Join us for a paper discussion on "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling"
Exploring tokenization-free language models through learnable hierarchical compression
Featured Paper:
"Dynamic Chunking for End-to-End Hierarchical Sequence Modeling" (Hwang, Wang, Gu, 2025)
arXiv Paper | Code
Discussion Topics:
H-Net Architecture Design

• Hierarchical U-Net structure with dynamic content-dependent chunking
• Encoder-decoder networks using Mamba-2 SSM layers for compression efficiency
• Main network operates on progressively compressed sequences (L_S << L_0)

Dynamic Chunking Mechanism

• Routing module predicts boundaries via similarity scores between adjacent elements
• Smoothing module interpolates representations to handle uncertain boundaries
• Ratio loss prevents trivial solutions (excessive compression or no compression)

Implementation Challenges

• Gradient-based learning of discrete boundary decisions
• Balancing compression ratio vs information retention
• Memory management for hierarchical processing stages

Key Technical Features

• End-to-end learning eliminates fixed tokenization biases
• Character-level robustness without special data augmentation
• Learned boundaries reveal semantically coherent chunks

Future Directions

• Multi-stage hierarchy for deeper abstraction levels
• Extension to multimodal data (audio, vision)
• Integration with existing transformer architectures

Silicon Valley Generative AI has two meeting formats:
1. Paper Reading - Every second week we meet to discuss machine learning papers. This is a collaboration between Silicon Valley Generative AI and Boulder Data Science.
2. Talks - Once a month we meet to have someone present on a topic related to generative AI. Speakers can range from industry leaders, researchers, startup founders, subject matter experts and those with an interest in a topic and would like to share. Topics vary from technical to business focused. They can be on how the latest in generative models work and how they can be used, applications and adoption of generative AI, demos of projects and startup pitches or legal and ethical topics. The talks are meant to be inclusive and for a more general audience compared to the paper readings.

If you would like to be a speaker or suggest a paper email us @ svb.ai.paper.suggestions@gmail.com or join our new discord !!!