Curating LLM data – Hamel’s Blog - Hamel Husain
Hamel Husain рассказывает, почему стоит создавать собственные инструменты для разметки и курирования данных при работе с LLM, а не полагаться на готовые вендорские решения. Он перепробовал Spacy Prodigy, Argilla и Lilac, но столкнулся с ограничениями: либо сложно адаптировать под конкретные задачи, либо не хватает базовых функций сортировки и фильтрации. Автор подчёркивает, что великие AI-исследователи (как Andrej Karpathy с ImageNet) лично смотрят на данные и строят инфраструктуру для быстрой ручной инспекции. После сравнения Gradio, Panel, Streamlit и Shiny For Python он выбрал последний за нативную интеграцию с Quarto, реактивную модель, компактный API и поддержку WASM. По его опыту, приложения на Shiny требуют меньше кода и понятнее, чем альтернативы.
TLDR
I think many people should build their own data annotation/curation tools for LLMs. The benefits far outweigh costs in many situations, especially when using general-purpose front-end frameworks. It’s too critical of a task to outsource without careful consideration. Furthermore, you don’t want to get constrained by the limitations of a vendor’s tool early on.
I recommend using Shiny For Python for reasons discussed here. I wouldn’t recommend Streamlit for reasons discussed here.
Background
One pattern I noticed is that great AI researchers are willing to manually inspect lots of data. And more than that, they build infrastructure that allows them to manually inspect data quickly. Though not glamorous, manually examining data gives valuable intuitions about the problem. The canonical example here is Andrej Karpathy doing the ImageNet 2000-way classification task himself.
I couldn’t agree with Jason more. I don’t think people look at their data enough. Building your own tools so you can quickly sort through and curate your data is one of the highest-impact activities you can do when working with LLMs. Looking at and curating your own data is critical for both evaluation and fine-tuning.
Things I tried
At the outset, I tried to avoid building something myself. I tried the following vendors who provide tools for data curation/review:
Vendors
These tools are at varying levels of maturity. I interacted with the developers on all these products, and they were super responsive, kind and aware of these limitations. I expect that these tools will improve significantly over time.
One thing that became clear to me while trying these vendors is the importance of being able to hack these tools to fit your specific needs. Every company you work with will have an idiosyncratic tech stack and tools that you might want to integrate into this data annotation tool. This led me to build my own tools using general-purpose frameworks.
General Purpose Frameworks
Python has really great front-end frameworks that are easy to use like Gradio or Panel and Streamlit. There is a new kid on the block, Shiny For Python, was my favorite after evaluating all of them.
Reasons I liked Shiny the most:
I found that Shiny apps always required much less code and were easier to understand than the other frameworks.