Building an AI Second Brain: Architecture Deep Dive.

Briefing

An AI second brain is not a chatbot that remembers things. It is a system that captures, structures, compresses, and retrieves your digital life at scale. OpenHuman and its Memory Trees are, by the maker's own framing, one of the most complete consumer attempts at this so far, but the underlying ideas work in any knowledge management setup.

Analysis

Picture the part of your job that nobody pays you for: remembering where the decision was made. Which Slack thread, which email, whose meeting note, which pull request. You know the answer existed. You just cannot find it when it matters. That gap is what a second brain is built to close.

The pitch is simple to say and hard to do. Pull in everything you touch across your tools, give it structure, and let software hand the right piece back to you at the right moment. Less digging, fewer dropped commitments, and a running picture of where your week actually went.

OpenHuman, an open-source local-first project from TinyHumans AI, is the version of this you can run right now. It connects to your accounts, watches what is on your screen if you let it, and keeps a memory file you can open and read yourself. The thing worth understanding is not the product. It is the shape underneath it, because once you see the six layers, you can spot a real second brain from a glorified note-taker.

The Architecture

A second brain has six layers:

Ingestion -> Processing -> Storage -> Retrieval -> Synthesis -> Action

Layer 1: Ingestion

Ingestion captures data from every source you care about. OpenHuman ships with 118+ third-party integrations that auto-fetch every 20 minutes, covering GitHub, GitLab, Linear, Notion, Slack, Discord, Gmail, Google Calendar, and more. The desktop mascot adds screen intelligence: what you are reading, what code you are writing, what documents you have open.

Ingestion has to be:

• Comprehensive: Missing sources create blind spots
• Real-time: Stale data is less useful than fresh data
• Non-intrusive: It must not get in the way of your work
• Respectful: You control what gets captured and what does not

Layer 2: Processing

Raw data is useless without structure. Processing normalises, enriches, and connects:

• Normalisation: GitHub issues, Linear tickets, and Notion tasks become the same fundamental entity type
• Entity extraction: People, projects, technologies, and deadlines are identified and linked
• Relationship mapping: "This email references that GitHub issue which relates to this Linear ticket"
• Sentiment analysis: Flagging urgency, frustration, and excitement

interface ProcessedEntity {
  id: string;
  source: IntegrationSource;
  type: EntityType;
  content: string; // Markdown
  entities: ExtractedEntity[];
  relationships: Relationship[];
  sentiment: SentimentScore;
  embeddings: Float32Array;
  timestamp: Date;
}

Layer 3: Storage

OpenHuman's Neocortex is marketed as storing up to 1 billion tokens locally, though the Neocortex repo's own spec cites accurate handling of over 10 million tokens, so treat the billion-token headline as a marketing figure rather than a benchmarked one. The author describes the storage as a hybrid approach:

• BM25: Exact text matching for precise retrieval
• Dense embeddings: Semantic similarity for conceptual search
• Graph traversal: Relationship queries ("what did Sarah say about the API?")
• Hierarchical compression: Recent data is detailed; old data is summarised

Worth a caveat here: OpenHuman's docs confirm the hierarchical summary trees and on-device embeddings, but the BM25 and graph-traversal mechanisms are this article's characterisation rather than documented features. The Memory Trees format itself is an Obsidian-style Markdown wiki: human-readable, version-controllable, and portable. You own your data. It lives on your machine, not in someone else's cloud.

Layer 4: Retrieval

Retrieval has to be fast, relevant, and contextual. The query "what was the decision about authentication?" should return:

1. The decision document from DECISIONS.md
2. Related Slack discussions
3. The GitHub PR that implemented it
4. Follow-up Linear tickets
5. Your own notes from the meeting

Retrieval quality is measured by MRR (Mean Reciprocal Rank) and NDCG (Normalised Discounted Cumulative Gain). As an illustrative benchmark, a well-tuned second brain might aim for MRR > 0.7 on common queries, though that threshold is an author's figure rather than a published OpenHuman or industry number.

Layer 5: Synthesis

Synthesis is the part that earns its keep. The second brain does not just find information, it generates insights:

• Weekly reflections: "You spent 40% of your time on billing this week"
• Connection surfacing: "This problem resembles one you solved three months ago"
• Priority triage: "Three deadlines are approaching; here is the optimal order"
• Knowledge gaps: "You have been mentioned in 12 threads but have not responded to 5"

OpenHuman's Subconscious loop handles synthesis continuously in the background, working away while you stop typing. The cadence is reportedly hourly merges, daily summaries, and weekly reflections, though that exact schedule was not confirmed verbatim in the documentation, and it runs without you having to ask.

Layer 6: Action

The whole point of a second brain is action. Information that does not change what you do is just trivia:

• Inline autocomplete: Suggesting relevant code based on your research
• Meeting preparation: Summarising context before a Google Meet
• Follow-up reminders: Surfacing commitments you have forgotten
• Cross-reference linking: Automatically linking related documents

Building Your Own

You do not need OpenHuman to build a second brain. The architecture can be implemented with open-source tools:

Ingestion: n8n or Huginn for integrations
Processing: Python scripts with spaCy/Transformers
Storage: SQLite + FTS5 + pgvector (or Chroma for embeddings)
Retrieval: Custom search with hybrid ranking
Synthesis: Periodic LLM calls (Claude API, OpenRouter)
Action: Webhooks to your tools

These are all real, widely used tools, so the stack is sound. But the integration effort is substantial. OpenHuman's value is not the architecture, it is the execution: 118+ integrations working out of the box, a polished desktop app, and the Subconscious loop running continuously without configuration.

Privacy Considerations

A second brain knows everything. That is its power and its risk:

• OpenHuman stores everything locally. No cloud required.
• Screen intelligence processes on-device. Screenshots never leave your machine.
• Third-party integrations use OAuth, and the tokens are reportedly held in the macOS/Windows keychain, though that specific storage detail was not confirmed in the documentation.
• Markdown exports let you inspect exactly what is stored.

The local-first model is a deliberate privacy choice. A cloud-based second brain would be more powerful (unlimited storage, always available, team sharing) but would mean trusting a third party with your entire digital life. OpenHuman bets that most users will pick privacy over convenience.

The second brain is not a settled product category yet. It is an aspiration. OpenHuman is the closest implementation I have seen, and it ships updates often, with the project moving past v0.57 by mid-2026. The author predicts that within 12 months "second brain" will be as common a term as "code editor", which is a forecast worth noting rather than a sure thing.