In today's thrilling episode, we dissect “LongNet”, a groundbreaking paper that scales transformers to a whopping 1 billion tokens. Next, we discuss Uncertainty Alignment and its implications for robotics. Finally, we cover "Motion Retargeting", a method of creating 3D avatars from minimal user input data, primarily headset and controller information.
Key Points
1️⃣ LongNet
A method called "LongNet" scales transformer models to handle a billion tokens, using dilated attention to avoid quadratic complexity, achieving linear scaling.
While this method technically handles a billion tokens, it's different as it looks at pieces, not the entire attention, compromising performance beyond context window.
It's viewed as a clever innovation in computational scaling, despite trade-offs, and other methods like 'alibi' are suggested for better performance.
2️⃣ Uncertainty Alignment
The paper introduces "uncertainty alignment," a method for robots to handle ambiguous tasks by seeking minimum user help and providing statistical guarantees before executing a task.
This approach reduces fine-tuning and prompt tuning, aligns with how people think, and improves user experience by asking follow-up questions when uncertain.
While not groundbreaking, it simplifies complex tasks using probability and statistics, potentially becoming a standard practice for various chatbots and robotics applications.
3️⃣ Motion Retargeting
“Motion retargeting" is a method of creating 3D avatars from minimal user input data, primarily headset and controller information.
This technology transfers human movements to various virtual characters, demonstrating realistic movements despite the difference in character structure, like a dinosaur or a mouse.
Though promising, the technique depends heavily on the user's movements, and edge cases like extreme physical behavior can disrupt the avatar's realistic representation.
🔗 Episode Links
Connect With Us:
Follow us on Threads
Subscribe to our Substack
Follow us on Twitter:
LongNet | Uncertainty Alignment | Motion Retargeting