What We're Building
Ok so most likely you skipped through that preface pretty quick. That's ok that's fine, let's talk about what we're actually building.
This is a local MacOS (sorry windows / linux users) Tauri desktop application. It should be ~150MB. It has numerous features but within it, it holds a local agent that knows how to interact and search your iMessages. This is not sniffing the Ollama space and this is not meant to be a bring-your-own-local-model application. BRB is meant to give you a better understanding into your relationship with your peers, show interesting texting patterns and analytics, and overall just be a helpful tool for visualizing and self-reflection. That's really what I want - people to be able to answer questions about their lives in a safe and secure way, gain better insights, and have fun using this application.
Core Features
We have these as the features I want to highlight:
- local-first: all data and processing happens on your mac. no cloud, no external servers, messages stay private
- iMessage integration: indexes and understands your iMessages, enabling advanced search, semantic queries, natural language Q&A
- convo analytics: visualize your text patterns as the illustrious Github Activity heatmap, message freq, response times, etc etc
- casper agent: interact with an agent that can help you answer daily questions
- performant: lightweight application, downloads weights on the fly, won't burn out your vram
- secure: again, your data is your data
Architecture
System Diagram
Here I have embedded an excalidraw diagram showing the overall system. I'll be doing this throughout.
We'll get into this a lot more
Key Design Principles
These were / are the basic tenets that I wanted to ensure survives for this project:
1. local first. data does not leave your machine.
Your data never goes over the network. The only sockets in this project are unix domain sockets - there's no TLS nor TCP.
I have bricked my computer building this more times than I can count running / experimenting with this. I try to take some preemptive measures about protecting against OOMs given your unified memory constraints. For initial ingestion and for fine-tuning, I'll eat your RAM up rather quickly. You can configure various settings to go against this.
2. grounded responses from your data
Yeah yeah, I know RAG is dead and has been dying for about three years. We have long context!! 1M context for Opus 4.6/Opus 5 - yes. I get it and I know.
It's a hot button topic in the engineering community, and there are many many many great posts about it. Those links are borderline chronological.
Regardless, I use vector embeddings and RAG. And I use it in a very fun way through DuckDB's VSS extension which i'll be talking a lot about later on. Plus, this project has some other nifty bells and whistles. There's hybrid BM25 search plus a custom reranker! The functional approach for people is that it's actually referencing chunks of your iMessage data and the hallucinations should be minimal.
3. self-reflection over simulation
I am trying to show people grounded in their data insights that they might not have seen before. It's legitimately not that far off from what I am trying to do with Odyssey. With regards to fine-tuning on someone else's messages, that's not allowed. You can create a digital clone of yourself and chat with it, but you cannot do that for others. Consent needs to be guaranteed and that is an engineering + design element that I do not have.
And concretely: if Casper doesn't have data on something, it will say so - not play along. If you ever catch it doing otherwise, file a Github issue and I'll take a look.