Sources & Methods

The Archive Playbook

How to process a person's life archive, start to finish.

The Archive Playbook

How to process a person’s life archive, start to finish

Created 2026-06-02, distilled from Fred’s project (letters, journals, email, Facebook, Quora, Yelp, photos, videos). This is the sequence — what to do, in what order, and why. For the how-to of each method (OCR, Whisper, mbox extraction, LR catalog queries, model selection), see sources-and-methods.md, the field guide this playbook points into.

The core lesson from doing it the unsystematic way: the disciplines below all emerged mid-stream, after the cost of not having them was already paid. Format blockers surfaced one source at a time. people.md got re-reconciled across letters → Facebook → photos because there was no registry first. The pipeline layers were named only after narrative work stalled for lack of a self-analysis layer. Next person, front-load all of it. That’s the whole point of this document.

The order of operations

Do these phases roughly in order. Phases 0–2 are cheap, fast, and save weeks — never skip them to “get to the good part.” The temptation is always to jump straight to reading the interesting letters; resist it until the map exists.

Phase 0 — Inventory & triage map (before touching any content)

Build the status table first, as the starting artifact — not derived at the end. (This project built it at rev 5, after most of the work. Backwards.)

List every source: physical and digital, located and rumored. Walk the drives, the cloud, the closets.
For each: size/count, date span, format, where it physically lives, and access cost (iCloud now / SSD at home / physical retrieval / conversion service).
Assign each a pipeline stage (○ not started … ✓ synthesized) and a rough value/effort estimate.
Output = the status table (see project-status.md, regenerable via the project-status skill). This is your map for the whole project.

Discipline: inventory before you dig. A comprehensive inventory before assuming you know what you have. Platform fragmentation is the rule, not the exception — journals alone spanned Newton, Word for Mac, and Apple Notes.

Phase 1 — Format triage (decide the blockers early)

Before processing, answer one question for every source: what format is this, and what converts it? Do this as one front-loaded pass, not one stall at a time.

Sort each source into: readable now / convertible (tool known) / at-risk (readable but format dying) / blocked (needs emulator/service).
Prioritize at-risk formats first — the recovery window closes. DV tape, old word-processor files (WDBNMSWD/TEXTMWII), firewire camera tapes, DAT all follow the same obsolescence curve. A one-afternoon ffmpeg batch now beats an unrecoverable file later.
- WDBNMSWD (MacWrite) → mwaw2text: brew install libmwaw, then mwaw2text "filename" > out.txt. Text extraction only — formatting ignorable. Supports MacWrite 4.5/II/Pro; handles endianness. Fred’s pre-1996 archive (1987–1990) is this format but unrecoverable — backup captured resource fork only, not data fork. Text is gone.
Flag conversion-service items (DAT tapes, firewire capture) early so they can run in parallel with everything else — they have long lead times and cost money.

Phase 2 — Scaffolding (set up once, reuse everywhere)

Stand these up before processing the first source:

Directory convention — one top-level dir per source type, each with _work/{_source,_scripts,_sessions,_transcripts} and root-level INDEX/README/handoff. Copy small sources in; symlink large ones. (Full spec in the field guide.)
People registry — people/source/people.md, one entry per person, sources tagged inline. Start it on day one. It gets mined from every source type repeatedly; building it late means re-reconciling the same names across letters, email, FB, and photos. Add an entry the moment a name first appears; flesh it out as you read.
Personal-history timeline — _reference/personal-history.md, organized by life domain (geography, work, relationships, vehicles), not strict chronology — easier to update incrementally.
Citation convention — establish it now. Retrofitting citations is labor-intensive and this project is still backfilling tags created before the convention existed. (See self-analysis/citation-conventions.md.)

Phase 3 — Process each source through the pipeline

For each source, run the four-layer pipeline. Each layer cites the layer below it, never the raw source. Full provenance model: workflows/provenance-pipeline.md.

The source layer is the durable asset. Targets (wiki, book, EPUB) are renderings of the source layer — ephemeral and regenerable. Build the source records right; the target follows from them. You can wipe any target and regenerate from source, or produce an entirely different target, because the materials are preserved in source/ and processed/.

Directory convention within each source type:

source/ — raw originals, never modified
processed/ — normalized/extracted working layer; points back to source
target/ — output (wiki articles, etc.); points back to processed via source.file
archive/ — dead ends, superseded files, unreadable formats; investigated and done

Layer organization: provenance in, logical-unit out. The standardization step (source/ → processed/) is where source-location stops driving structure. (Generalized from journals, 2026-06-25.)

source/ is organized by provenance — one subfolder per original medium/location (4TB Archive/, Apple Notes/, LiveJournal/, Gmail/). Raw formats differ wildly and provenance governs trust + scrub depth, so keep staging grouped by where it came from. (This is the “medium as source/<medium>/ subfolder” rule above.)
processed/ and target/ are organized by the source’s natural logical unit — not by provenance. The unit differs by source-type: journals/letters/email → by date; notes/photos/recipes → by subject or event. Once a thing is standardized, every consumer (chunking, wiki, analysis) cares about when / what, not where-from. Provenance is demoted to frontmatter metadata — never lost. The payoff is a single unified timeline/subject-space across all media.

Standard frontmatter on every standardized .md (the metadata that carries provenance forward once it leaves its source/<medium>/ folder):

date: 2011-03-04          # sort key for dated sources (or `subject:` for note/topical sources)
source: apple-notes       # provenance tag — <medium> or <medium>/<era>
source_path: "…"          # resolvable path/URL to the original (anchors the provenance chain)
visibility: admin         # public | login | admin — default admin until scrubbed

source_path is the provenance-chain anchor from the discipline below: a derivative without it is an orphan. visibility uses the wikifred model (lib/visibility.mjs).

Multi-source collisions — when one date/subject carries entries from two media (a journal day and a LiveJournal post; a letter and an email): keep them as separate files tagged by source, merge only at target/ time. Pre-merging loses the provenance distinction and invites chops.

If no source exists, create it. An article without a source record is a prompt to create one — from memory, research, GEDCOM, photo metadata, whatever is known. Memory-sourced portraits → people/source/portraits/. Photo stubs → photo-reviews.jsonl. The article and source record are co-authored; neither should exist without the other.

Swim upstream, spawn downstream. Start at source when generating content. Enumerate source entries → check which are claimed → create articles for unclaimed ones. Never start from articles and try to find their sources retroactively (one-time fix only).

Triage      sort, prioritize, decide what to process and when
   ↓
Discovery   transcribe / identify / read — "what is this?" → facts
   ↓
Self-analysis  aggregate facts into thematic structured docs
   ↓
Narrative   autobiography chapters, portraits — drawn from self-analysis

Capture facts immediately. When discovery surfaces a name, date, or event, write it to people.md / personal-history.md in that session — don’t strand it in session notes waiting for an analysis pass that may never come. Good discovery nearly eliminates the need for a separate analysis pass.
Don’t skip self-analysis. The pull from discovery straight to narrative is strong; the self-analysis layer is what makes a 60-year narrative coherent.
Use the minimal viable model. Haiku subagents for batch extraction/triage; Sonnet for vision (OCR, photo ID); Whisper (local) for audio/video; Sonnet/Opus only for synthesis. (Model-selection table in the field guide.)
Resumable scripts for anything >20 min. Design batch jobs to skip already-processed files, start them, let them run overnight, collect results next session. AI sessions don’t persist; scripts do.

Phase 4 — Synthesis & narrative

Draw from the self-analysis docs to write autobiography chapters and portraits. This is where judgment and voice live — worth the expensive model and the human pass.

Cross-cutting disciplines

These apply at every phase. They are Fred’s logged corrections — bake them into the method, don’t rediscover them per batch.

Second pass by default. After completing any triage batch, do a slightly deeper second pass. It reliably finds what the first missed.
Err slightly too deep. When triaging, missing content is worse than a few extra reads. Depth over shallow.
Update the handoff at the start of a task, not just the end — it marks active work in progress if the session is interrupted.
Interpret by medium, not just by content. Each communication regime (letters / email / social media / AI) preserves a different slice and carries a different bias. A two-month gap in letters may mean phone calls; absence from Facebook is not absence from living. (See “The Archaeology of Communication” in the field guide.)
Index for the update cycle. A living person’s archive is never finished. A dated file-level index makes “what’s new since last pass” a query, not a re-scan.
Every source needs a defined extraction routine — not ad-hoc AI generation. Each data source (journals, letters, photos, Facebook, …) gets a documented, repeatable routine for turning source into articles, living either in the source project or in the presentation layer (e.g. WikiFred). Ad-hoc one-shot AI generation produced this project’s worst defects: the “Monterey chop” (a journal entry with multiple dated sub-entries split into two articles, the first keeping the whole entry → a duplicated tail), fabricated detail (invented birthdays, a fictional mechanic), and rimshots. A routine must: segment cleanly (one journal sub-entry → one article; split on the dated YYMMDD.HHMM markers, no overlap), link the source, and preserve the source’s own words. Ad-hoc generation is a one-time bootstrap; the routine is what makes it correct and re-runnable. Three corollaries:
- Surgical corrections are welcome. Once you learn a fact, edit the published article directly — add a known name + wiki-link, fix the source attribution. (E.g. “my friend Rebecca” → [[rebecca-casey|Rebecca]] once photos confirmed Rebecca Casey.)
- One source can hold several stories. It’s fine to chop part of a journal entry into an article — but then (a) don’t duplicate that part into a sibling article (the Monterey tail), and (b) don’t miss another distinct story sitting in the same entry.
- Cross-reference matches both ways. When a story↔photo (or story↔source) match is found, record it in BOTH places — the article frontmatter and the source/photo database — so the link is durable and queryable (“which photos still have no story?”).
Provenance chain: original → processed → derivative. Keep the original — in the archive if it fits, linked externally (drive path / URL) if it’s massive. Every derivative links back to the original, not just the processed copy it was made from. The failure mode (hit on the wiki): articles linked to the processed intermediate, so getting back to the source now requires a search instead of a click. A derivative without a resolvable path to its original is an orphan — you can’t re-check it, and garbles can’t be traced. (See the WikiFred Pinzgauer correction: paraphrased prose with a source: label but no source link.)
Sensitive content goes in a dedicated section, never inline. Any processed doc (portrait, analysis, transcript) that contains sensitive material — private relationships, medical, financial, gender expression, others’ private info — keeps that content in a clearly marked ## Personal / do-not-publish section at the end, isolated from the publishable narrative. Cut the line from the body; move it there. This makes the “what can ship?” question a section boundary, not a sentence-by-sentence hunt.
Context pass before item pass. Before processing a batch of items (photos, letters, a journal year), do a context pass first: read the background docs, establish who/where/when for the period, note any known events or people active at that time. Then process items. Context-first identification is dramatically more accurate than item-by-item in isolation — one photo identified correctly anchors the rest of the roll.
Sort by source-type for processing; label by medium for provenance. A single medium can hold multiple source-types, and they route to different pipelines. Every personal archive has at least two buckets, processed differently:
- Journals — dated, first-person, reflective prose → processed chronologically, mined for narrative/voice. The unit is the entry.
- Notes — topical, utilitarian, reference (collections, lists, logistics, specs) → processed by subject/extraction, rarely chronological. The unit is the subject. One medium (e.g. Apple Notes) commonly carries both — split it by type into the right project bucket, and keep the medium name as the _work/source/<medium>/ subfolder so provenance survives the split. The same medium-name can legitimately appear under two project buckets. (Hand-scanned pages will split the same way: some journal, some note.)
Search the big files; don’t load them into context. Hints, groupings, registries, and the per-item databases (e.g. people-photo-hints.md, photo-groups.md, photo-reviews.jsonl, reference/*, journal years) grow large. Grep for the matching entry rather than reading the whole file — usually you only need those lines. Tell subagents to grep, not read wholesale. Read whole only when genuinely necessary. (2026-06-08)
Cross-domain lookup — it’s all one life. When a fact isn’t in the directory you’re in or in reference/, look across other domains (people, photos, vehicles, letters, journals). The corpus is deeply interconnected; the right fact often lives in another domain’s files. Don’t give up after the local dir. (2026-06-08)
Canonical home per fact-kind — usually the richest source’s domain, not a generic reference/. Each kind of cross-cutting fact gets ONE home, and it belongs with whichever domain produces and resolves it. Photos are the richest source for both, so: date questions → photos/date-questions.md (every entry comes from photo review, resolved against the LR catalog), event/trip groupings → photos/photo-groups.md. Don’t default these to reference/ just because other domains might read them — put them where they’re produced. One event commonly spans several camera file-sequences (iPhone IMG_ + Fuji DSCF…, sometimes three) — a group unites them; it is not one contiguous filename run. (2026-06-08; date-questions moved photos-local per Fred.)
Cross-flow is owned by the TARGET, not the producer. When material from one artifact should enrich another (photo stories → a person/pet portrait; a letter → a portrait), the destination pulls it at build/refresh time (e.g. via enrich-portraits, which scans the photo corpus). Don’t push at production time — collect provisionally (e.g. production-notes.md) and let the target gather its own on a sweep. Keeps producers simple and gives each target one place that owns its completeness. (2026-06-08)

The oral-history exception (time-sensitive — do not defer)

Everything above is about material that already exists and will keep existing. Living memory will not. Family and friends hold knowledge that exists nowhere in the written record, and that window closes permanently. Prioritize interviews ahead of archive processing where there’s any risk — the SSD will still be there next year; the person may not.

Prep contextual questions-for-NAME.md before a visit; scan all sources first so questions are informed and you’re not carrying the library.
Verify every factual premise in a question; cut anything the subject can answer trivially or that you could answer yourself.

Quick-start checklist (new person, day one)

[ ] Walk every drive/cloud/closet → list all sources
[ ] Build the status table (Phase 0)
[ ] Format-triage every source; flag at-risk + conversion-service items (Phase 1)
[ ] Stand up directory convention, people.md, personal-history.md, citation convention (Phase 2)
[ ] Identify living people to interview → schedule before processing (oral-history window)
[ ] Pick the first source by value/effort → run the pipeline (Phase 3)

This playbook is the procedure; sources-and-methods.md is the methods reference; project-status.md is the live state. Expand this as the method matures — it’s a starting point, like everything else here.