LLMs: A Data Engineer's Game

We have a controversial opinion, but please hear us out: LLMs has become a data engineering problem rather than a data science problem! In this session we'll explore how data engineering teams can leverage LLMs with ease within the Databricks Data+AI platform. We'll explore how we can utilise commercial LLMs and build-your-own LLMs to bring natural language querying to your data. Best of both worlds: fine-tuned LLMs + commercial LLMs. We'll look at fine-tuning and applying Hugging Face models thanks to the ease of MLflow's integration with Hugging Face. Using SQL AI Functions to apply OpenAI GPT-3.5 text capabilities to unstructured data Call to action: We hope we demonstrate that it's possible for DEs to leverage LLMs today with a combination of fine-tuned LLMs and commercial LLMs. You don't need to wait for your data science team to start creating value from your data