LLMs as Capable Regressors & Optimizers
🔮 A game of peek-a-boo with data... Now you see it, now you don’t!
Welcome to AI Fridays, this week’s roundup of theÂ
latest innovations and breakthroughs in AI brought by HackerPulse in collaboration with AIModels.fyi.
🥇 Symbolic AI Geometry Smackdown: Wu's Method vs. IMO Medalists - Who’ll be left with angles intact?
🌀 The Curse of Recursion: Training on Generated Data Makes Models Forget
📊 No Zero-Shot Without Exponential DataÂ
🔢 From Words to Numbers: Your LLM Is a Capable RegressorÂ
🧠LLMs as Optimizers
Symbolic AI to Rival Geometry IMO Medalists (🔗 Read Paper)
Wu's Method is a technique that significantly enhances the performance of symbolic AI systems, allowing them to compete with silver medalists in the International Mathematical Olympiad (IMO) for geometry problems. Wu's Method combined with the deep learning system AlphaGeometry results in a hybrid approach surpassing the performance of even gold medalists at the IMO in geometry. This innovative combination showcases the potential of integrating classic symbolic reasoning with advanced neural network capabilities to tackle complex mathematical challenges.
The Curse of Recursion: Training on Generated Data Makes Models Forget (🔗 Read Paper)
The paper introduces the concept of "Model Collapse," which occurs when large language models (LLMs) like GPT-3 and GPT-4 are trained on content previously generated by other models. The model-generated content training across different model types and datasets leads to a loss of diversity and the disappearance of unique or rare elements from the original data. This issue can impact a variety of generative models, including Variational Autoencoders and Gaussian Mixture Models.
The research sheds light on a critical challenge in the training of LLMs and other generative models. It highlights the need for prioritizing genuine human interactions in training data to prevent Model Collapse and sustain the benefits of large-scale data training.
No Zero-Shot Without Exponential Data (🔗 Read Paper)Â
The paper investigates how the frequency of concepts in pre-training data affects multimodal models' zero-shot performance. It reveals that models trained on data with a long-tailed distribution of concept frequencies struggle to learn rare concepts, limiting their zero-shot capabilities. The study suggests exponential growth in pre-training data is needed for models to achieve strong zero-shot performance across a diverse range of concepts. This highlights the challenges faced by models in effectively generalizing to new tasks and underlines the need for vast amounts of diverse data to enhance model performance.
From Words to Numbers: Your LLM Is Secretly a Capable Regressor (🔗 Read Paper)Â
This paper explores the hidden ability of large language models (LLMs) to perform regression tasks when given in-context examples. LLMs can effectively convert text prompts into numerical outputs, showcasing their potential as versatile regression models. Moreover, LLMs like GPT-4 and Claude 3 can rival or even outperform traditional regression models when given in-context examples. For instance, Claude 3 outperformed several supervised methods on the challenging Friedman #2 regression dataset. This discovery expands the application of LLMs beyond language tasks, presenting new opportunities for their use in various fields such as finance, science, and engineering.
LLMs as Optimizers (🔗 Read Paper)Â
This paper introduces "Optimization by PROmpting" (OPRO), a novel approach that leverages large language models (LLMs) as optimizers by describing optimization tasks in natural language. Traditional gradient-based methods have limitations when gradients are unavailable, but OPRO offers an effective alternative. It iteratively generates new candidate solutions from the prompt, evaluates them, and adds the best ones to the prompt for the next optimization step. The paper showcases OPRO's effectiveness across various optimization problems, including linear regression, the traveling salesman problem, and prompt optimization for language models. OPRO significantly outperforms human-designed prompts on tasks like GSM8K and Big-Bench Hard, showcasing the potential of LLMs as versatile and powerful optimization tools.Â
Ever been told the secret to their success is simply "keep working"?
Fran has been there, chatting with Senior and Principal Engineers at Amazon. It frustrated him not knowing how to replicate their success.
That's why he decided to dive deep into career growth and find actionable tips everyone can apply. No more vague advice.
And that’s a wrap. Talk soon! 👋
Awesome, love it!