ClassifAI Studio
ClassifAI Studio is a Mac app for asking questions about your data. Run several models on the same questions, compare their answers, and check them against labels you trust.
Every model gets the same questions and gives answers in the same shape, so you can put a small local model next to a large LLM and compare them directly.
They read the input and the questions and return a probability for every option, taken from the model itself. They are fast and cheap, and their probabilities are usually well calibrated.
ClassifAI Studio turns your questions into a JSON schema, so an LLM answers in the same format. It also quotes the part of the input it used, and it can answer open questions. Its probabilities are its own estimates.
Run a dataset through several models and score each question against its ground truth. You get accuracy, F1, Cohen's κ, calibration and latency per model, a confusion matrix, and a calibration chart that shows whether a model's confidence means anything.
A review queue shows the answers most worth checking. When you correct a label there, the scores update without running the models again.
Go through a dataset entry by entry, mostly from the keyboard. Select a passage, or hold ⌥ and move by sentence, and press ↵ to keep it as the evidence for an answer. Each question has its own colour, so you can see which passage supports which answer.
Your labels become a dataset you can evaluate models against.
Turn your labels into a model of your own, trained on your Mac's GPU. Set it up like an evaluation, pick a base model, and ClassifAI holds back rows for the test, measures the models you compare against, trains, calibrates, and puts the fine-tune next to them in one leaderboard.
You see each step explained while it runs, and afterwards what it learned: per label, how far to trust its confidence, and the rows it now gets right. The result works everywhere in the app like any other model, and stays on your Mac.
Connect hosted decision APIs and LLM providers next to your local models. Search a provider's model list to add a model, and its published prices are filled in. You see what each model costs per entry and can set a monthly budget. API keys stay in the macOS Keychain.
The parts you need around the models.
Pick one, pick any, yes/no, scale, rank, number and free text, each with its own context and option descriptions.
Ask a question only for certain answers, such as "if No, ask why". Models answer it in a second pass with the earlier answer added.
For a decision model, parts of the input are left out one at a time to see what moved the answer. For an LLM, the quoted passage is found in the input.
Answer the same questions for many entries with one model, then save the answers as a dataset or CSV.
CSV, JSON, Excel, the Hugging Face Hub, and folders of Word files, PDFs and images.
Export an evaluation as Markdown or HTML. The classifai command line works on the same data as the app.
Open a row from an evaluation or a fine-tune in the Playground: its questions and ground truth come along, and each answer is marked right or wrong.
What decision models, evaluations and fine-tuning are and how to read the numbers, from the data split to LoRA and calibration, with a glossary.
Download the latest build for Apple Silicon Macs from the releases page.
The app is not notarised by Apple yet. The first time you open it, macOS will refuse. Right-click the app and choose Open, or on macOS 15 and later allow it under System Settings → Privacy & Security → Open Anyway. You can also run xattr -dr com.apple.quarantine "/Applications/ClassifAI Studio.app" in Terminal.
To build it yourself you need macOS, Rust, Node 20 or later and pnpm.
git clone https://github.com/jonasweinert/classifai-studio.git cd classifai-studio pnpm install pnpm tauri build
On first launch the app offers to install the Ollaya engine into its own folder and to download a small local model. You can skip that and use only cloud models or LLMs.