Vision-Language Club - Controlling temporal dynamics in Text-to-Video Models

After a long break, for the next meeting of the Vision-Language Club we are very excited to host Shira Schiber, for her talk: TempoControl: Controlling temporal dynamics in Text-to-Video Models

We will meet online, on May 4th at 20:00. The talk will also be recorded and uploaded to NLP-IL's YouTube channel. The talk will be conducted in Hebrew.

While Text-to-Video models have shown remarkable generative capabilities, they often lack fine-grained temporal control. Users cannot specify when a subject should appear or when an action should occur. In this talk, I will present TempoControl, a method that allows for temporal alignment of visual concepts during inference, without requiring retraining or additional supervision. I will explain how TempoControl utilizes cross-attention maps, a key component of text-to-video diffusion models, to guide the timing of concepts through a novel optimization approach. I will talk about the three complementary components of the loss function: correlation for aligning the temporal pattern with a control signal, magnitude for adjusting the strength, and entropy for preserving semantic consistency. Finally, I will demonstrate the various applications of TempoControl, including temporal reordering of single and multiple objects, action timing, and audio-aligned video generation.

Shira holds an MSc in Computer Science from Bar-Ilan University, where she specialized in video diffusion models under the supervision of Dr. Ofir Lindenbaum and Dr. Idan Schwartz. Her industry experience includes developing real-time models for avatar generation at D-ID, and training classification and segmentation models for semiconductor manufacturing at NI.

### Call for Speakers:
Are you working on something exciting in the field of NLP or Vision-Language and eager to share it with the community? We’re looking for future speakers! Apply here to give a lecture at one of our upcoming meetups: bit.ly/nlp_il_talk