WikiFred

Sources & Methods

Sources & Methods Field Guide

Tools and techniques for each source type.

Sources and Methods

How to Reconstruct a Life History from Digital Archives

“Digital archives” means the working state of the material, not its original form. Letters, journals, Super 8 film, DV tape, and photographs all begin as physical or analog objects — but every source passes through digitization before it can be processed. Scanning, OCR, telecine, video capture: whatever the method, the paper or tape becomes a file. This guide covers that whole arc, from original medium to processed text to synthesized narrative.

Owner: Fred Heald Created: 2026-05-27 Purpose: High-level map of all source material and processing methods for the autobiography/archive project.

Secondary purpose — recipe book: This document is intended to be generalizable. The specific sources are Fred’s, but the methods and pipeline should be usable by anyone undertaking a similar digital-plus-physical life archive project. Where possible, methods are written to transfer. Fred’s project may be unusually large in volume, but the structure — raw source → processed file → synthesis → autobiography — applies broadly.

Editorial note: This document is currently written at Fred’s level — it describes what was done, with project-specific paths and context. A publication pass should lower it one level: add example prompts a reader could use to adapt each method to their own data (e.g. “Create a script similar to extract-mbox.py to handle my AOL email export”). Each method section should end with at least one copy-paste-ready prompt.

On AI session boundaries and overnight jobs: Claude works interactively within a session but doesn’t persist between them. The practical pattern for long-running work: design scripts to be resumable (skip already-processed files), start them in a terminal, let them run overnight, pick up the results in the next session. The transcript pipeline (transcribe_lr.sh, cleanup.sh, process_transcripts.sh) follows this pattern deliberately. Claude designs the scripts and interprets the results; the machine does the compute. This division generalizes: anything that takes more than 20 minutes of wall-clock time should be a resumable script, not an interactive session.

Publication intent: This document (and the methods it describes) is a candidate for publication as a standalone guide — potentially on GitHub — for anyone wanting to undertake a similar life-archive project with AI assistance. Examples drawn from Fred’s work would be anonymized or chosen for safety/cleanliness. The guide would cover: digital archaeology of old formats, photo archive triage, email and social media triage, video transcription, and AI-assisted synthesis into autobiography. Working title: The Personal Archive Handbook or similar. Not started — capture methods fully first.

Scope

This project draws on ~60 years of primary source material: Fred’s own writing across multiple platforms, family documents, personal correspondence, and physical archives. The goal is an autobiography (possibly multi-volume) plus a family history component, with subsidiary products: portrait documents, self-analysis docs, journal entries.

Directory Structure

Every source directory follows a consistent three-part layout:

<source>/
  README.md, CONVENTIONS.md, handoff.md   ← navigation layer
  source/        ← files as arrived (exports, JSONLs, PDFs, transcripts.jsonl)
    <subdir>/    ← categorized as needed
  processed/     ← human-navigable output (digests, transcripts, per-day chunks)
    <subdir>/    ← categorized as needed
  workflows/     ← process docs, scripts, how-tos

Large originals (exports, binary files) live on the 4TB Archive at /Volumes/4tb Archive/bups/platform-exports/<source>/; the repo’s source/ is either a lightweight staging area or a README pointing there.

Cross-domain analysis lives in self-analysis/ — not per-source analysis/ dirs.

Subdirectories within source/ and processed/ are used as needed — e.g. videos/source/transcripts/ (raw whisper output) and videos/processed/transcripts/ (cleaned) + videos/processed/summaries/.

Source Corpus

Journals vs. notes — the basic division of a personal archive. Two kinds of self-authored material, distinguished by what they are, not where they’re stored:

  • Journals — diaries, life-experience observations, reflections, dreams, travelogue. First-person and experiential; the record of a life as lived and felt. Processed chronologically, mined for narrative and voice; the unit is the entry.
  • Notes — all the other documents one gathers: reference material, lists, specs, collection logs, how-tos, work scratch, logistics. Useful but not reflective. Processed by subject/extraction; the unit is the topic.

A single medium can hold both (Apple Notes carried a 260-entry journal corpus and ~1,283 reference notes; hand-scanned pages will split the same way). Sort by this type to route into the right project bucket — journals → journals/, notes → notes/ — and keep the medium name as the _work/source/<medium>/ subfolder so provenance survives the split. See the playbook’s cross-cutting disciplines for the processing side of this rule.

Cross-cutting workflows

Workflow What it covers
workflows/provenance-pipeline.md Source provenance chain — how source/processed/target layers relate, source.file convention, retroactive linking, feedback loop for missing source records. Start here if an article has no source.
playbook.md End-to-end sequence — inventory → triage → pipeline → synthesis. Provenance theory in Phase 3.
wikifred.md Wiki presentation layer — article format, voice rules, tagging, scrub rules.

Per-source workflows

Each source has a detailed workflow doc in its own dir — acquire/capture → extract → synthesize → articleize, with the source-specific rules that bite. The type-based subsections below describe what each source is; the workflow docs describe how to work it. Shared article-creation rules live in wikifred.md.

Source Workflow doc Wikifred src- thread
Journals ../journals/journals-workflow.md src-journal
Yelp ../yelp/yelp-workflow.md src-yelp
Quora ../quora/quora-workflow.md src-quora
Photos ../photos/photos-workflow.md (assets — attach to articles)
Facebook ../facebook/facebook-workflow.md src-facebook
Blog ../blogs/blogs-workflow.md src-blog
Autobiography drafts ../autobiography/autobiography-workflow.md src-autobiography

Personal Journals

Source Coverage Format Status
Apple Notes journals 2013–2026, 264 entries Digital text Read; synthesized into self-analysis docs
Archive journals (year files) 1999–2012, ~12 files, ~8,933 lines Markdown Extracted to journals-archive/; partial analysis
Physical journals 1991–1996 (28 volumes), then 1996–1999 (1 volume) Paper, in storage Not digitized; requires hand-scanning each page/fold. Critical gap — 1992–1996 digital originals are largely unrecoverable (TEXTMSWD), so physical journals are the only first-person voice for that period.

Personal Correspondence

Source Coverage Format Status
Scanned letter PDFs 1965–2000s, ~228 PDFs PDF (vision OCR) Active — see letters/INDEX.md
Recovered digital originals 1985–1998, ~120+ files Old Mac word processor → text Partially extracted; see letters/sources/text/
Gmail export 2001–2026, 36,566 messages MBOX Extracted to mbox-archive/; not yet triaged
Local mbox archive 2001–2009, 9,772 messages MBOX Extracted to mbox-archive/; not yet triaged
Instant messages (AIM, iChat, etc.) ~1995–2010s Varies Not yet located or extracted
SMS/iMessage Ongoing Apple Messages DB Not yet extracted

Social Media & Public Writing

Source Coverage Format Status
Facebook archive (JSON) 2007–2026, 13,348 posts, 10,899 comments, 1,396 message threads JSON Extracted 2026-05-26; posts split into year + topic files. Posts synthesized 2026-05-27 → self-analysis/facebook-notes-YYYY.md (2007–2026); comments + messages not yet triaged
Quora (fheald account) ~1,862 entries Exported HTML Fully mined 2026-06-02; quora/_processed/mining-notes.md; autobiography drafts + self-analysis written
Quora (yudro account) 2017–2026 Exported See reference/yudro.md
Obtainium blog 2002–2006, 207 posts SQL dump Extracted to obtainium-archive/blog/

Books, Movies & Media

Source Coverage Format Status
Goodreads export Reading history CSV/analysis books-movies/ — analyzed; see goodreads_analysis.md
Music analysis Listening history / preferences Analysis books-movies/music_analysis.md
Media recommendations — Analysis books-movies/media_recommendations.md

Note: books-movies/ (formerly personal-media-analysis/) contains Fred’s aesthetic record — reading history, music taste, media preferences. Useful for autobiography chapters on cultural identity and taste formation. Not a pipeline project; contents are analysis outputs, not raw source material.

Family Archives

Source Coverage Format Status
Phil’s Mac backup (SSD) ~2000–2025, ~50K files Mac backup Actively researched; see projects/pheald/
Fred’s 4TB Archive drive 1985–2012+ Mixed Partially read; year archives copied to letters/sources/text/
Gramp’s computer backup 2002, pre-death Mac backup Partially read; research papers + Villa Rose journals
GEDCOM files Heald.ged, Clark.ged Genealogy Read; see reference/family-history-sources.md
Gram’s travel journals 1960s Physical Held by Fred (Russia) and Kathie (world trip 1962)
AncestryDNA raw data 2016 SNP text Not yet analyzed
IHCO photo archive 1908–2008 TIFF scans Inventoried; not yet described

Photo Archives

Source Coverage Format Status
Lightroom catalog (Extreme SSD 2TB) 1922–2026, ~99,894 photos SQLite catalog + JPEG/RAW Primary repository. Queryable — keywords, collections, dates, face data. See below.
Desktop Pictures 4+ export 1983–2000 (mostly), 1,538 images JPEG Derived subset — 4+ star photos exported from LR, not a separate collection. Active review project enriching LR photos with stories and identification; 190/1,538 done — photos/.
Facebook media 2007–2026, ~300+ images JPEG In Facebook export; linked via ![photo]() in posts.md
Phil’s Australia slides (1966) 1966 Scanned Held by Fred

Lightroom catalog as a primary historical source (expanded 2026-05-30)

Location: /Volumes/Extreme SSD 2TB/Pictures/Fred Photos/Lightroom Catalog-2-2-v13-3-2.lrcat
Format: SQLite database — query directly with sqlite3 "file:[path]?mode=ro" "SELECT ..."
Scale: ~99,894 photos, 1922–2026. Well-organized with named collections, people tags, and some captions.

What the catalog contains:

Data type Table(s) Value
Collections AgLibraryCollection, AgLibraryCollectionImage Named events with date-prefix names (e.g., 1993-07-04 Boston, 000903_BMK2K). The single best gap-year chronology source.
People tags AgLibraryKeyword (keywordType=‘person’), AgLibraryKeywordImage Named individuals with first/last photo dates and photo counts. Confirms when people entered and exited documented periods.
Capture dates Adobe_images.captureTime ISO timestamps on all photos; queryable by year/month/day.
Captions AgLibraryIPTC.caption Manually entered on some photos; sparse but substantive where present.
Camera models AgHarvestedExifMetadata, AgInternedExifCameraModel Equipment succession over time; useful for dating undated photos by camera era.
Other-photographer photos Collections named _otherpeoples, _mariepierre, etc.; keyword Other People's Fred imported other people’s photos; these are tagged by source, not subject.
Face detection AgLibraryFace, AgLibraryKeywordFace LR face detection applied to the full library; “Unnamed Face” = 9,551 entries. Useful for cross-reference but not reliable on its own — LR misidentifies.

Use cases discovered 2026-05-30:

  1. Gap-year chronology — Collection names are the most complete record of what Fred was doing in 2000–2004. Before this discovery, those years were largely undocumented. Collection names confirmed: Boston Dooffest (July 4, 1993), BM years 1998–2004, Maui (June 2002), Florida (Feb 1997), and dozens of other events with no other source.

  2. People timeline — Query first/last photo date per person to see when they entered Fred’s documented world. Confirmed: Becke first appears September 2001 (BM 2001); Pamarie first appears April 2004; Sally Thornhill from September 2001.

  3. Resolving photo dates — Cross-reference an undated scan against LR collections by matching visual content, people, and era. See photos/date-questions.md for open cases and reference/lr-journal-crossref.md for established links.

  4. Face reference set — When a Desktop Photos review photo has an uncertain ID, query LR for confirmed photos of the suspected person from the same period. Example: 12 Pamarie-tagged photos from BM 2006 serve as a face-reference set.

  5. Camera dating — Camera model in EXIF data helps date undated scanned photos by exclusion. E950 era = 2000–2001; E4500 era = 2002–2004; D70 era = 2005+.

Cross-reference approach (Desktop Photos vs. LR digital): No filename overlap — Desktop Pictures were scanned 2025 with sequential scanner names. Match by: identifying the event in a review session (“this looks like BM 2003”) → query LR collection → use resulting photos as reference.

Photo export from catalog — orientation correction (2026-06-20):
sips does not apply LR catalog rotation. After exporting, read Adobe_images.orientation and apply correction:

LR value Meaning sips --rotate
AB normal none
BC 90° CW 90
CD 180° 180
DA 90° CCW 270

Exception — JPEG source files: sips does apply the camera’s EXIF orientation when re-encoding JPEG→JPEG (it rotates pixels and clears the EXIF tag). LR’s catalog orientation often mirrors the original EXIF, so applying it again produces a double-rotation. Symptom: portrait shots exported correctly by sips end up sideways after the correction script. Fix: apply catalog rotation only for RAW source files (CR2, NEF, ARW, DNG, etc.), not for JPEG originals from cameras where sips handles it automatically.

Evenly-spaced photo selection: when a collection has more 4★ photos than the max (10), pick evenly spaced: step = (len-1)/(n-1), use {round(i*step) for i in range(n)} as indices. Produces representative coverage without clustering at start/end.

Audio Recordings

Source Coverage Format Status
Voicemails (saved) Ongoing Audio files (m4a, mp3, wav) Not yet collected or transcribed
Recorded conversations Ongoing Audio files Not yet collected or transcribed
Phone/video call recordings Ongoing Audio files Not yet collected or transcribed

These are high-value biographical sources — a saved voicemail from a parent, a recorded conversation with a family member, captures voice and speech patterns that no written source preserves. Entirely Whisper-compatible: extract audio, run transcription, result is a timestamped text primary source.

For a generic user: saved voicemails on a phone can be exported via apps like Google Voice export, Visual Voicemail backup tools, or direct file access on some Android devices. iOS voicemails can be shared from the Phone app. Once as an audio file, the pipeline is identical to video transcription.

Next step for this project: collect any saved voicemails or call recordings into audio/ (a directory not yet created); transcribe with the same whisper-cli pipeline.

Video Archives

Source Coverage Format Status
Videos Archive (4TB Archive drive) 1994–2025, 1,856 files MOV/MP4/AVI/DV/MTS Indexed 2026-06-01 → videos/videos-index.jsonl

Folder breakdown: Dylan Phone Movies (679), _Dylan (492), _pets (197), Quadcopter Movies (150), Burning Man Movies (142), _misc (82), Jack Hyatt (39), Hanen Videos 2015 (19), _diaries (19), _unsorted (11), Chloe’s Wedding (9), Heald Movies (8), 2008 Movies (6), Diane’s Tattoo (2), 940630 Mister H’s.mpg (root — San Pedro dinner party, June 30, 1994).

What the index contains: path, filename, folder, date (metadata or filename-derived), duration, size, codec, resolution, inferred who/where/description, bio_flag. The notes and when fields are blank — available for manual annotation as files are reviewed.

Content gap: The index is metadata-only. Video content — what was said, who appears on screen, what happened — is not yet described. This is the largest gap between this source type and others. See Methods below.

What we can do now:

  • Metadata extraction — date, duration, codec, resolution via mdls/ffprobe. Reliable and fast (done).
  • Frame extraction — ffmpeg -ss [time] -vframes 1 pulls a still from any point. Useful for identifying content, confirming faces, or picking representative images.
  • Filename/folder inference — who, where, and occasion inferred from path. Accuracy is high for named folders (_Dylan, _pets) and low for generic camera-roll names (IMG_XXXX).
  • Audio transcription — Whisper (OpenAI, open-source, runs locally) can transcribe speech from any video file. This works now. Accuracy is good for clear speech; lower for ambient noise, overlapping voices, or non-English.
  • Format assessment and conversion — ffmpeg batch-converts at-risk codecs to H.264. Runnable as a one-time cleanup job.

What’s coming (near-future, possibly already available):

  • Transcript extraction at scale — Whisper or equivalent over the full 1,856-file corpus. Would create a searchable text layer for every video that has speech. For the Hanen therapy sessions, Dylan diaries, and video journals, this would be transformative — the content would become as searchable as journal text.
  • Frame-by-frame description — Vision models can describe video frames in sequence. A scene-change-detected frame grid per video gives a rough visual summary without playing the file. Already possible with ffmpeg (scene detection) + Claude vision.
  • Person identification in frames — Cross-reference extracted frames against LR face set. Not reliable autonomously, but AI can narrow candidates for human confirmation.
  • Content summarization — Once transcripts exist, summarize what was said in a 30-minute home video in 3 sentences. This is a standard LLM task once the transcript is available.
  • Search across the corpus — With transcripts + descriptions, “find all videos where Dylan mentions Stick Fluffy” becomes a grep. Without them, that information is locked in the files.

Recommended next steps (priority order):

  1. Run Whisper over Hanen Videos 2015 — Done 2026-06-01. 19 files transcribed to videos/hanen-transcripts/. Content: Becke + therapist Masala working with Dylan age ~1. Ball rolling, Little Blue Truck book, shape sorting, letter recognition, songs. Directly biographical — documents Dylan’s developmental baseline.
  2. Run Whisper over _diaries — Done 2026-06-01. 18/19 transcribed (1 corrupt file). Fred’s own Sanyo camera video diaries (~2009–2014); largest files 39 and 41 minutes. Output: videos/diaries-transcripts/.
  3. Frame-extract key files in _pets for Dora, Pasquale biography
  4. Batch-convert the 37 at-risk codec files (see Format Risk below)
  5. Review Jack Hyatt folder (39 files, unknown content)

Lessons Learned: Bulk Video Transcription (2026-06-01)

Transcribing video at scale is possible and the pipeline works well — but a phone camera roll is not a diary. Running Whisper over everything produces a lot of noise. Target specific videos rather than bulk-transcribing a whole library.

What you’ll find in a camera roll:

  • Live Photos — Every iPhone photo since ~2015 also generates a 2–3 second MOV file. These are the dominant file type in most libraries (78% of LightroomMasters was ≤3s). No useful audio. Filter by duration before transcribing.
  • TV/YouTube audio — Phone left on while something plays on TV, or Dylan recording YouTube. Transcribes as convincing but completely irrelevant text (The Voice, game commentary, etc.). Hard to auto-detect; catch during reading pass.
  • Game audio — Same problem; often produces coherent-sounding but context-free dialog.
  • Drone footage — DJI files (prefix DJI_): even when audio is present, it’s propeller noise. Skip entirely.
  • Hallucination loops — When Whisper encounters silence or unclear audio, it sometimes latches onto one phrase and repeats it dozens of times. Detectable: many lines, ≤3 unique.
  • Blank audio / annotations only — [BLANK_AUDIO], (upbeat music), etc. Auto-deletable.

What survives and is worth it:

  • Intentionally filmed conversations, family moments, birthday parties
  • Events (weddings, celebrations) — the Kip/Anett wedding ceremony, 2018, was captured in full
  • Video diary entries — Fred’s Sanyo camera diaries (2008–2010) are the richest source in the collection
  • Anything where someone is clearly speaking to the camera or to another named person

Practical filters (in order):

  1. Duration > 3s — eliminates Live Photos
  2. Skip DJI_ prefix — eliminates drone/Osmo footage
  3. Post-transcription cleanup: delete single-line results, annotation-only files, and hallucination loops (>5 lines, ≤3 unique)
  4. Reading pass: discard TV/game audio manually

Signal-to-noise in practice: In LightroomMasters (6,319 files), roughly 20% survived duration filtering, and of those transcripts, roughly 30–40% contained genuinely useful biographical content. The effort is worth it for the gems — but go in knowing that most of what you get back is the ambient noise of a life, not the signal.

Source link convention: Each transcript should have a # source: /path/to/original.MOV header line so the video is recoverable. Add # people: Name, Name when you know who’s on screen. These are point-in-time path references — valid at time of transcription, may not survive file moves.

Scripts: videos/_work2/index_lr.sh, transcribe_lr.sh, cleanup.sh, process_transcripts.sh

Format risk:

Codec / Format Count Risk Action
H.264 (MOV/MP4) 1,686 None Safe for decades
MPEG-1 Video 20 Low Readable everywhere; convert eventually
MPEG-2 Video 8 Low DVD-era; readable
DV (.dv files) 35 Medium Camcorder format, increasingly orphaned; batch-convert to H.264
H.263 1 High 1990s video conferencing codec; convert now
Apple Intermediate Codec 1 Medium Apple-proprietary edit codec; convert now

Full format analysis: videos/videos-index.md

The broader format-obsolescence note in the Challenges section below applies here: DV in particular is following the same trajectory as WDBNMSWD word processor files — currently readable, but the window is not infinite. The DV batch conversion is a one-afternoon ffmpeg job.

Generalizability to other photo libraries:

This approach works with any photo management application that uses a SQLite catalog:

  • Apple Photos (iPhoto/Photos.app): ~/Pictures/Photos Library.photoslibrary/database/Photos.sqlite — contains similar tables for albums (≈ collections), people (face clusters), keywords, and EXIF data. Schema differs but same SQLite approach applies.
  • Google Photos (via Takeout): No local SQLite, but the metadata JSON files exported with each photo contain album membership, timestamps, and description fields — queryable with a simple script.
  • digiKam: Uses SQLite (digikam4.db) with similar collection/tag/face structure.
  • Capture One: Catalog is SQLite-based.

The general principle: any photo library software that organizes photos into named albums/collections with dates is a chronological event log, not just a photo viewer. The organization the user did at import time is historical data. Query it as such.

Methods

PDF Vision OCR (scanned letters)

Tool: Claude Sonnet (vision API) via subagent
Process: Agent reads each PDF image, transcribes verbatim, writes structured markdown to letters/_transcripts/. Physical description noted, uncertain readings marked [word?].
Quality: Sonnet used (not Haiku) for handwritten/typewritten letter accuracy.
Output format: See letters/CONVENTIONS.md

Old Mac Word Processor Recovery

Tool: strings (resource fork), textutil -convert txt, LibreOffice
Recoverable: CDFV2, RTF, plain text, resource-fork ASCII
Unrecoverable (without Classic Mac emulator): WDBNMSWD, TEXTMSWD, TEXTMWII, TEXTMACA
Notes: Most 1992–1997 files are in unrecoverable formats. 1985–1991 largely recoverable.

Archive Extraction Scripts

Generic scripts (any user of that platform would need these) live in sources-and-methods/workflows/scripts/:

Script Purpose Status
extract-facebook.py Facebook JSON archive → year files + topic files ✅
extract-obtainium-sql.py SQL blog dump → post files ✅

Type-specific scripts live in each data type’s _work/workflows/scripts/:

Script Home Purpose Status
extract-mbox.py email/workflows/scripts/ MBOX → monthly text files ✅
extract-archive-journals.sh journals/workflows/scripts/ Year archives → _work/archive/ ✅
organize-letters.py letters/workflows/scripts/ Letter file organization ✅

iPhone Backup Extraction — voicemails, texts & attachments (added 2026-06-27)

iPhone backups are a goldmine almost everyone already has and overlooks. A local Finder/iTunes backup (~/Library/Application Support/MobileSync/Backup/<UDID>/) contains the device’s visual voicemail (Library/Voicemail/voicemail.db + .amr recordings) and all SMS/iMessage (Library/SMS/sms.db

  • attachments), fully extractable offline — no third-party app, nothing leaves the machine.

How the backup is laid out. Files aren’t stored by name; Manifest.db (a SQLite index) maps each original path to a hashed fileID, and the actual file lives at backup/<fileID[:2]>/<fileID>. So the recipe is always: read Manifest.db → find the path you want → copy the hashed file out.

Voicemails. In voicemail.db, the voicemail table has ROWID, sender, date (unixtime), duration, trashed_date; each recording is Library/Voicemail/<ROWID>.amr. Rename to Voicemail from <sender> on <date>.amr and transcribe (see Audio Recordings — use mlx-whisper, ~1.4s/file on Apple Silicon, ~12× faster than reloading whisper.cpp per file).

Texts. In sms.db: message (text, handle_id, date in nanoseconds since 2001-01-01, is_from_me, service, cache_roomnames for groups), joined to handle (the phone/email) and to attachment via message_attachment_join. Newer rows store text in attributedBody (a streamtyped blob) when text is NULL — decode best-effort, and preserve the raw blob (base64) for anything that fails so nothing is lost. Attachments are MediaDomain files under Library/SMS/Attachments/....

Caller/contact names come from the Mac’s own Apple Contacts — the per-source AddressBook DBs ~/Library/Application Support/AddressBook/Sources/*/AddressBook-v22.abcddb (join ZABCDPHONENUMBER→ ZABCDRECORD; the top-level db is empty — names live in the iCloud source dbs). Normalize numbers to 10 digits.

Encrypted backups. If MDM/work management (or the user) forces “Encrypt local backup,” the manifest is locked behind the backup password. Decrypt offline with the iOSbackup Python package + the password. Encryption changes nothing else about the method.

Two caveats that shape what you get:

  • Visual voicemail only retains undeleted messages, so each backup is a trailing window, not a full timeline. Extract every backup you can find (old devices, Time Machine, external drives) and merge — each may cover a different span.
  • “Messages in iCloud,” when on, moves the text store to the cloud and out of local backups (a backup will then show only a handful of messages). Older backups predating that setting hold the full local history — they’re the goldmine. iCloud-resident messages can’t be read off-disk; recovering them means restoring to a device or a forensic downloader.

Worked result (Fred’s archive): one unencrypted iPhone 6s Plus backup yielded 44,993 messages (2008–2020), 968 contacts, 2,307 attached media and 118 voicemails (2019–2020) — including 7 more recordings of his late father’s voice that existed nowhere else.

Storage convention (where the artifacts go): by-year working originals (.amr, messages-<YYYY>.jsonl, attachments) → Documents Archive/<YYYY>_archive/{voicemails,texts}/. The cross-year source-of-source artifacts — the raw device backup itself (the true original) plus the extracted master DBs (sms.db, voicemail.db) — → bups/device-backups/<device>/ (one folder per device; the raw backup is otherwise single-copy on the Mac’s internal drive). Scripts: extract-texts.py, transcribe-mlx.py (resumable). See biography/voicemails/README.md.

⚠️ Lesson (learned the hard way, 2026-06-27): back up before you wipe — or give away — any device. Fred recovered a decade of his own voicemails and 100k+ texts from old iPhone backups. But his late father Phil’s old phone was wiped and handed down (no backup) and the backup computer was wiped — real losses. The single rule that would have saved those: make a local backup of any phone/computer before erasing, selling, gifting, or letting an account lapse. But the story turned hopeful: Phil’s new iPhone, backed up and decrypted, carried a synced catalog of 12,639 photos back to 2001 — proof his iCloud (an ancient @me.com / MobileMe-era account, alive since ~2008) was still live. ~2,500 full-res photos came straight out of the backup (3.3 GB); the other ~10,000 sit in iCloud, retrievable by toggling “Download and Keep Originals” — without signing out. Lessons within the lesson: (a) confirm an account’s real status before assuming loss (Fred had only feared it was terminated); (b) a device signed into a live iCloud is itself a recovery vector — never sign out until you’ve pulled the data down. Corollary for archivists: the people around you hold the other half of your history — have family back up their devices now, while they exist. (iOS 26 backups may omit Info.plist, which trips iOSbackup; work around it with a symlinked working dir + a stub Info.plist — never edit the original backup.)

Reusable prompt: “I have an iPhone backup at ~/Library/Application Support/MobileSync/Backup/. Write an offline Python script that reads Manifest.db, extracts the voicemails (voicemail.db + .amr) and all texts (sms.db + attachments) — resolving sender names from my Apple Contacts AddressBook DBs — and writes them out organized by year. If the backup is encrypted, use the iOSbackup package with my backup password. Flag whether ‘Messages in iCloud’ looks enabled (very few messages in the store).”

AI Model Selection — Use Minimal Viable Model

A general principle for this project and any similar life-archive project:

Use the smallest model that can do the job. Heavy batch processing should run as subagents using the cheapest capable model — typically Haiku. Reserve larger models (Sonnet, Opus) for synthesis, judgment calls, and work that requires nuance.

Task type Model Reason
Batch journal analysis (extract facts by category) Haiku Text-only, repetitive structure, high volume
Letter transcription (handwritten/typewritten OCR) Sonnet Vision required; accuracy matters; handwriting is hard
Photo description (faces, context, content) Sonnet Vision required; identification needs reliability
Video transcription whisper-cli (local, not Claude) Not an LLM task — dedicated ASR tool is faster and cheaper
Synthesis (autobiography drafting, portrait writing) Sonnet or Opus Judgment, voice, nuance — worth the cost
Quick lookups, file operations, shell scripts Haiku No reasoning required
Triage passes (email, social media, large corpora) Haiku Pattern recognition at scale; human reviews output

Subagent pattern: Spawn a Haiku subagent with a batch of files and a structured output format. Collect results. Review in the main context. This keeps the main conversation context clean and makes large-scale processing practical.

Example prompt for a Haiku subagent:

“Read these 10 journal files. For each, extract: dates mentioned, people named, places, key events, emotional tone (one word). Output as JSON, one object per file.”

For a generic user: the same principle applies regardless of archive size. Even a small personal archive benefits from separating cheap batch extraction from expensive synthesis — the extraction creates the raw material; the synthesis is where the meaning gets made.

Discovery: AI Journal Analysis (Haiku subagents)

Tool: Claude Haiku (cost-efficient, text-only)
Process: Agent reads year journal files, extracts biographical facts by category (geography, work, relationships, vehicles, key events), writes to self-analysis/journal-notes-YYYY.md
Completed: 1999, 2002, 2004
Pending: 2000, 2001, 2003, 2005, 2006, 2007+

Photo search for a person — two tiers

Default (cheap): search the processed photo records first, every time — photos/photo-reviews.jsonl (reviewed favorites, with who/description/lr_people/lr_tags). Covers the curated set.

Expensive option: query the full Lightroom catalog (…/Lightroom Catalog-*.lrcat — a SQLite DB on the archive SSD) only when the processed records come up empty for someone and you specifically need their images. This relies on the photos having been face-/keyword-tagged in Lightroom — untagged photos won’t surface no matter how many exist.

  • Find the keyword: SELECT id_local,name FROM AgLibraryKeyword WHERE name LIKE '%Name%';
  • Gauge volume (some people have dozens — “you’ll get too many”): filter keepers by rating, … AND ai.rating=4.
  • Resolve full paths: AgLibraryKeywordImage → Adobe_images → AgLibraryFile → AgLibraryFolder → AgLibraryRootFolder (absolutePath || pathFromRoot || idx_filename).
  • Copy candidates locally and view them to choose (capture dates are often the scan date, not the photo date).
  • Apply the young/old/action selection; skip group shots.
  • Cost: the SSD must be mounted; read-only queries; dozens of images + visual review per person — deliberate, not default.

The Pipeline

Source material (physical or digital)
    ↓  Triage        sort, prioritize, decide what to process and when
    ↓  Discovery     identify, annotate, surface facts — "what is this?"
    ↓  Self-analysis aggregate discoveries into structured thematic docs
    ↓  Narrative     autobiography chapters, portrait docs

Triage — decide what to work on. Sort sources by value and effort. Some things (email corpora, large photo sets) need triage before any other step. Output: a prioritized queue and rough notes on what’s there.

Discovery — look at a source and find things. Transcribe letters, identify photos, watch video clips, read journal passages. Output: transcriptions, annotations, photo-reviews.jsonl entries, session notes. Discovery yields facts — preserve them immediately in people.md and personal-history.md rather than letting them strand in session notes. This is the key discipline: discovery should almost eliminate the need for a separate analysis pass.

Self-analysis — aggregate what discovery found into thematic structured docs. self-analysis/relationships.md, self-analysis/burning-man.md, _reference/personal-history.md, people/source/people.md. Output: organized factual record ready to inform narrative.

Narrative — write the autobiography. Draw from self-analysis docs, not raw sources. Portrait docs, chapter drafts. Output: autobiography/ and people/source/portraits/.

Key principle: Each layer cites the layer below it, not the raw source. An autobiography chapter cites a processed file (e.g., britt-letters-1991.md), not the raw PDF scan. This keeps citations navigable.

For a generic user: the pipeline is the same regardless of archive size. Even a small personal project benefits from separating these steps. The temptation is to jump from discovery straight to narrative — but the self-analysis layer is what makes the narrative coherent across a life’s worth of material.

The Feedback Loop (source → article → back to source)

The pipeline is linear on paper; in practice it’s a loop. Per item (a person, a photo, a recipe):

_source/   RAW          the raw material — GEDCOM, scans, photo files, journals…
                        — plus intermediate cleanup that's still "getting raw into
                        usable shape" (scan → OCR; GEDCOM extraction). Many raws can
                        feed one item (gedcom + photos + letters + journals → a person).
   ↓ process
_target/   DATA RECORD  ONE cleaned, linked record per item. Knows about its sources —
                        cites/links back to them. Ugly is fine; it's working data.
   ↓ extract
article    OUTPUT       the wiki article, generated from the data record. Does NOT care
                        about sources — it works from prepared data.
   ↓ review
comment    FEEDBACK     the user reads the article and comments/corrects in the
                        per-article review box (→ review-questions.md).
   ↺ flows back to the top
                        Substantive corrections go to the data record (or to _source
                        if it's a raw-data fix) — NOT just the article. The article is
                        then re-derived. The loop closes; the record gets smarter each pass.

Feed corrections to the top, not the story. The article is a derivative; editing it directly desyncs it from its source and the fix is lost on the next regeneration. A fact correction (e.g. “a tractor accident killed him; the fistula was the hospital finishing it, not the cause”) belongs in the data record — which regenerates the article, so every future use draws the corrected fact. Prose polish (a trite opener, a clumsy sentence) can be fixed in the article itself; substance flows up.

Derivative vs. source. A per-person photo list (crosslinked from the photos source) is a derivative — regenerable, sits beside the data record, never hand-edited. The data record is the curated source-of-truth the article reads from. Keep the two distinct.

Naming standard: source/ = raw + intermediate · _target/ = the cleaned/linked data record · the article (in wikifred) is the output. (Older docs below still say _work/source/; _source/_target is the going-forward standard.)

Project Directory Structure Convention

Each source type (photos, videos, letters, etc.) has its own top-level directory. Within each:

[type]/
  _work/
    _source/      raw files, or symlinks to large files on external drives
    _scripts/     processing tools (Perl, Python, shell)
    _sessions/    review session logs
    _transcripts/ derived text output (whisper, OCR) — organized by source folder
  [type]-index.jsonl   master index
  handoff.md
  README.md

Copy vs. link: What goes in _work/source/ depends on size:

  • Small sources (letters, documents): copy the file — project is self-contained, readable offline
  • Large sources (videos, photos): symlink to the external drive path — project holds the derived work product (transcripts, annotations, JSONL) while source stays on the archive drive

The symlink approach means source/ always shows what was processed and where to find it, without duplicating hundreds of gigabytes. The project remains useful even when the drive is unmounted — transcripts and indexes are always available.

What lives at the root (not in _work/): stable outputs — the master index JSONL, README, conventions, handoff, TODO. These are the things you navigate to directly; _work/ is the machinery behind them.

Video Transcription (Speech-to-Text)

Tool: whisper-cli from whisper-cpp (Homebrew) — open-source, runs fully locally, no data leaves the machine.
Model: ~/whisper-models/ggml-base.en.bin (142MB, English-optimized). Larger model large-v3 (~3GB) available for higher accuracy if needed.
Pipeline:

video file (.MP4 / .MOV / .MTS / .AVI)
    ↓  ffmpeg -ar 16000 -ac 1 -c:a pcm_s16le
16kHz mono WAV
    ↓  whisper-cli -m ggml-base.en.bin
timestamped transcript (.txt)

Speed: ~5–6× realtime on M2. A 40-minute video takes ~8 minutes.
Quality: Good for clear speech; mishears on toddler speech, overlapping voices, strong ambient noise. Common pattern: “duck” → “fuck” when said by a toddler. Context resolves most errors.
Output format: One .txt per video, lines like [00:01:23.000 --> 00:01:25.000] text here. Functions as both transcript and time index.

Cross-source recovery: Video transcripts regularly surface biographical details that didn’t make it into written records — names, events, context mentioned in passing on camera that were never written down, or that appear in journals not yet analyzed. When a transcript yields a new fact, update people.md or personal-history.md immediately. Don’t wait for a full journal analysis pass that may never come.

What transcription unlocks:

  • Video diary entries become journal-equivalent primary sources — searchable, citable, usable in autobiography chapters exactly like written journals
  • Therapy recordings document developmental baseline and progress — Hanen sessions (2015) show Dylan age ~1 working on language, motor, book engagement
  • Family home videos yield biographical facts: who’s present, what’s said, what’s happening — equivalent to a journal entry for events that weren’t otherwise recorded
  • Phil’s DV tapes (once digitized) would add his voice narrating real estate and family events — potentially significant for family history

Completed:

  • Hanen Videos 2015/ — 19 files → videos/hanen-transcripts/
  • _diaries/ — in progress (2026-06-01) → videos/diaries-transcripts/

Reuse script: /tmp/diaries_transcribe.sh — change SOURCE_DIR and OUT_DIR to process any folder. Skips files already transcribed.

Generalizability: This pipeline works for any spoken-word video archive. The only requirements are ffmpeg and whisper-cpp (both via Homebrew). The base.en model is sufficient for most archival speech; large-v3 for difficult audio.

Provenance and Citations

All processed files, synthesis documents, and autobiography chapters should cite their sources using the conventions in self-analysis/citation-conventions.md.

Summary:

  • Autobiography chapters: formal footnotes → [^slug]: processed-file.md — date/item
  • Analysis/ files and portraits: inline source tags → *(src: filename.md · date)*
  • Portrait “Sources” lines: retained as high-level summary even when inline tags are present

Challenge: Retroactive citation is labor-intensive. The convention was established 2026-05-28; documents created before that date have Sources lines but not inline tags. Backfill during normal work — add tags when reading or updating a document, not as a dedicated pass.

Challenge: For large processed files (journal-archives, Facebook posts.md), locating a specific passage requires a stable identifier other than line numbers (which rot on edit). Convention: use date headers or short quoted search strings as locators. See self-analysis/citation-conventions.md for full guidance.

Coverage State (as of 2026-05-27)

Letters (scanned PDFs): ~40% transcribed

  • Done: 1965–1980 (partial), 1981–1984 Webb era, 1984–1988 Pomona era (partial), 1990 (~80%), 1991 (~60%)
  • In progress: 1992 not yet started
  • Not started: 1993–2026 (except scattered high-value items)

Recovered digital originals: ~60% read

  • Done: 1985–1991 most files
  • Partial: 1992, 1994, 1995, 1998
  • Blocked: 1996–1997 (TEXTMSWD/WDBNMSWD unrecoverable)

Journal analysis: ~15% processed

  • Done: 1999, 2002, 2004 notes files
  • Pending: 2000, 2001, 2003, 2005–2012 (all readable, not analyzed)
  • Not started: 2013–2026 (Apple Notes corpus — read but not formally analyzed)

Email archives: 0% — pipeline (not yet executed)

  1. Triage by sender — identify high-value correspondents (family, significant relationships, major life events)
  2. For each high-value thread: read, extract biographical facts → personal-history.md, relationship data → relationships.md
  3. Mine for stories → autobiography/drafts/; update people.md with new entries
  4. Low-value bulk (newsletters, work, spam): skip or summarize year-level only
  5. Source: email/source/ (organized by month); use grep or mbox reader to search by sender/date

Facebook comments + messages: 0% — pipeline (not yet executed)

  1. Comments: sample cross-section by year; note recurring themes, voice, relationships; add insights to self-analysis/ files
  2. Messages: prioritize long threads with significant people; extract relationship data → relationships.md; look for stories not captured elsewhere
  3. Source: facebook/source/messages/; already extracted

Yelp: 0% — pipeline (script not yet written)

  1. Write extraction script: parse yelp/source/Yelp/html/user_review.html → structured markdown (date, business, rating, review text)
  2. Organize by year → yelp/reviews/YYYY.md
  3. Mine for autobiography material: places Fred lived, aesthetic sensibility, recurring themes → yelp/analysis/yelp-notes.md
  4. Cross-reference with personal-history.md (confirm locations, timeline)

Apple Notes: partially analyzed — pipeline

  1. Source: notes/notes-index.jsonl (indexed); raw notes in Apple Notes / iCloud
  2. For each note: classify (journal/fragment, reference, todo, other)
  3. Journal-class notes → journals/originals/YYYY/MM/ appropriate file
  4. Reference notes → check if already in _reference/ docs; merge if not
  5. High-value fragments → fragments.md

Physical journals (1991–1994+): not started — pipeline

  1. Scan each volume — one-pass flatbed scan per page
  2. OCR with standard tool (tesseract or similar); create raw text file per volume
  3. Run journal analysis pipeline (same as archive journals): extract biographical facts by category
  4. Add to journals/_by-year/ structure; update personal-history.md
  5. Bottleneck: physical access required; in storage

GEDCOM genealogy: not started — pipeline

  1. Parse Heald.ged and Clark.ged with python-gedcom or similar
  2. Extract: names, dates, relationships → structured format
  3. Add key ancestors to people/source/people.md (flag: genealogy source)
  4. Cross-reference with existing personal-history.md family entries; fill gaps
  5. Bottleneck: 4TB Archive drive required

Facebook posts: Fully synthesized (2007–2026) — see self-analysis/facebook-notes-YYYY.md

Quora: Fully mined 2026-06-02 — see quora/_processed/mining-notes.md

Quora mining pipeline (repeatable):

  1. Read all topic files (autobiography.md, wit.md, vehicles.md, burning-man.md, aviation.md, food.md, technology.md, philosophy.md, politics.md)
  2. Create quora/_processed/mining-notes.md — categorize gems: autobiography stories, biographical facts, self-analysis material, wit/fragments
  3. Process each category to destination:
    • Stories → autobiography/drafts/[story-name].md
    • Facts → _reference/personal-history.md (BM, aviation, vehicles sections)
    • Self-analysis → appropriate self-analysis/ file with source citation
    • Wit/fragments → journals/originals/YYYY/MM/fragments.md
    • People → people/source/people.md
  4. Mark each item ✓/→ in mining-notes; update project-status.md

Facebook comments + messages / Obtainium: Extracted, not systematically mined

Ongoing Updates — Living Archives

For a living person, the archive is never finished. Journals, photos, videos, letters, and social media all accumulate continuously. Good indexing serves two purposes: it documents what’s there now, and it makes it easy to spot what’s new.

The update cycle:

Index built → new material arrives → diff against index → process new items

For each source type, “processing” means running the full pipeline: Triage → Discovery → Self-analysis → Narrative updates as warranted.

Practical triggers:

  • Journals: monthly — when a month is complete, check Apple Notes export against source/, import anything new, run discovery pass, update people.md and personal-history.md with new facts
  • Photos: batch — when a shoot or event is complete, run against LR catalog, add to review queue
  • Videos: as shot — import to archive, add to videos-index.jsonl, transcribe if speech-rich
  • Email: quarterly or after significant periods — triage new months from the mbox
  • Social media: annually — re-export and diff against prior export

Why indexing matters for updates: A file-level index (like videos-index.jsonl) with dates makes it trivial to find what’s been added since the last processing pass — query for records after the last-processed date. Without an index, every update requires re-scanning the whole archive.

“Process” as a term: Fred uses process to mean the full pipeline from raw input through triage, discovery, and self-analysis for a given batch of material. It’s not a single step — it’s the whole arc applied to a new batch.

Generalizable Reference Patterns

Two documents in this project are directly transferable to any life-archive project:

people/source/people.md — People Index A running who’s-who built from all sources as they’re read. One entry per person; sources cited inline (L=letters, J=journals, etc.). Grows organically — add an entry when a name first appears, flesh it out as more sources are read. Convention: when a blog, website, or social media profile is found for someone, add it to their entry.

For a generic user: start with the people who appear most often in your earliest sources and work outward. Don’t try to be comprehensive upfront — let it build from reading.

_reference/personal-history.md — Personal History A structured timeline of the subject’s own life: geography, career, relationships, vehicles, recurring themes. Distinct from the autobiography itself — this is the raw chronological record, not the narrative. Updated as new facts emerge from sources; the autobiography draws from it rather than replacing it.

For a generic user: organize by life domain (geography, work, relationships) rather than strictly chronologically — easier to update incrementally, and the patterns across domains become visible.

_reference/context.md — the lean context layer (two-tier architecture) (added 2026-06-03) people.md and personal-history.md grow into deep, narrative references — exhaustive and right for a human reading them, but too long and signal-diluted to feed an automated process that just needs to resolve “who is Tory?” or “what vehicle is in this 2008 photo?” The fix is a two-tier split:

  • Tier 1 — deep references (people.md, personal-history.md, portraits/): authoritative, exhaustive, grow freely. Update these when new facts emerge.
  • Tier 2 — context.md: a single lean file that combines the best of the Tier-1 files — compressed indexes (people one-liners, vehicles→years, places→when, master timeline, relationships) — for fast who/what/when/where lookups.

The rule for all data-source workflows (photos, videos, letters, journals, email, Facebook, Quora, etc.): do the context pass via context.md first. It self-documents its own escalation (“if it’s not here, see people.md / personal-history.md / portraits”), so source docs only need to point at context.md — don’t repeat the escalation chain in every workflow file.

Build mode: generated (preferred over hand-curation — eliminates drift). Because the deep files are prose, compression needs an LLM pass, not a script (“regenerate context.md from people.md + the personal-history tables”). Optional tiny hand-pinned top section, preserved across regenerations. Regenerate whenever the deep files change materially.

For a generic user: once your people index and personal history get past a few pages, add this lean generated layer so downstream tools read one small file instead of everything.

Directory Convention

Every source type (people, photos, letters, email, journals, etc.) follows the same structure:

<type>/
  README.md              ← orientation: what this type is, what lives here
  workflows/
    workflows.md         ← process documentation for this type
    scripts/             ← automation scripts that execute the workflows
  source/                ← raw/canonical data; never delete, only add
  target/                ← synthesized documents, generated on demand
  archive/               ← superseded material worth keeping

Key rules:

  • source/ is the ground truth. Raw inputs, data records, fetched references. Edits flow here first; everything downstream is generated from it.
  • target/ is lazy — create documents there only when something needs them (a WikiFred article, a research pass, a visit). Don’t pre-generate speculatively.
  • archive/ is anything superseded but not deletable. Old processing runs, prior versions, reference material that’s been incorporated elsewhere.
  • workflows/workflows.md is the primary process doc. If there are multiple distinct workflows, add them as additional files in workflows/.
  • scripts/ inside workflows/ — automation lives with the process documentation that defines it.
  • README.md at the type root is the entry point: what belongs here, what doesn’t, where to start.

Feedback loop: Edits and corrections in downstream products (WikiFred articles, synthesized docs) feed back into source/ as the authoritative correction. The pipeline is: source → synthesized (target) → presentation (WikiFred); corrections flow upstream.

On archive/: No strong rules on what goes in — anything you don’t want to delete but don’t want in the active directories. Superseded processing runs, old versions of canonical files, material that’s been incorporated elsewhere. The point is it’s still findable.

For a generic user: this structure applies to any archival type. Start with source/ and workflows/ — the others appear when you need them.

Key Reference Files

File Purpose
letters/INDEX.md Master PDF tracking: source → transcription
letters/CONVENTIONS.md Format guide for letter transcription sessions
reference/family-history-sources.md Deep inventory of family/genealogy sources
reference/4tb-archive-index.md Full index of 4TB Archive drive
self-analysis/relationships.md All people: who they are, connection to Fred
reference/context.md Lean context layer — start here for who/what/when/where (generated; points to the deep files)
reference/personal-history.md Deep ref: Fred’s biography — timeline, places, events
reference/people.md Deep ref: full people index (~280 entries)
reference/portraits/ Deepest dives — portrait docs
reference/chapter-concepts.md Emerging autobiography chapter hooks
self-analysis/citation-conventions.md How to cite sources; footnote format for autobiography chapters

Physical Journal Inventory (1991–1999)

Journals are in a box in storage. Hand-scanning required (every page or fold). Dates and cover descriptions as inventoried by Fred 2026-05-27.

Dates Location Cover
910101–910228 Aurora CO (First Journal)
910301–910414 Aurora CO —
910430–910504 Portugal Trip folding Japanese
910522–911010 Aurora CO —
911011–920109 San Francisco —
911031–911101/911227/920109–920331 San Francisco —
911111– San Francisco Clippings
920331–920506 Campbell CA orang
920507–920603 Campbell CA rhino
920603–920627 Campbell CA sharks
920628–920814 Campbell CA parrots
920814–921111 Los Angeles leaves
921111–930213 Los Angeles stripes
930213–930601 South Pasadena branches
930603–931116 — —
931116–940130 South Pasadena —
940130–940612 South Pasadena —
940612–941017 South Pasadena —
941017–941024/941027–941231 South Pasadena/Vista african print
950101–950218 Vista escher
950219–950508 Vista dk green
950508–950706/950807–950821 — homme du monde (dk green)
9507 Monterey (with Tory) —
960129–991124 — —
99 — Newton/misc computer?
00 — Misc computer
01 — Misc computer

Geographic corrections implied by journal dates (updates prior assumptions):

  • Fred left Aurora CO: ~May 22, 1991 (after Portugal trip)
  • San Francisco: Oct 1991 – Mar 1992
  • Campbell CA: Mar–Aug 1992 (Santa Clara County / Silicon Valley — not previously known)
  • Los Angeles: Aug–Nov 1992
  • South Pasadena: Nov 1992 onwards (consistent with Dec 1992 change-of-address card)
  • Vista CA: late 1994 – mid 1995
  • “Tory” = Victoria Paul: Monterey trip companion, July 1995

The Archaeology of Communication (added 2026-05-31)

This project spans roughly 60 years of Fred’s life across four or five distinct communication regimes. Each regime has a different preservation rate, a different social function, and requires different analytical methods. Understanding these differences is essential to interpreting the sources correctly.

The regimes

1. Telephone and paper letters (~1965–1995)

The dominant medium for personal communication. Letters were written with care, often revised, and preserved selectively — the recipient decided what to keep. Telephone calls left no record at all.

What survives: A biased sample. Letters that were saved were saved for a reason — importance, sentiment, or accident. The archive is richest where someone (Fred, Phil, Elise, Gramp) was a habitual keeper. What’s absent: every casual conversation, every phone call. The texture of daily life is almost entirely missing; what survives is the considered version.

Methods implication: Read letters for what was worth writing down, not for comprehensive coverage. The gaps are as significant as the text. A two-month silence in the correspondence may mean nothing (phone calls) or everything (estrangement).

2. Email (~1995–2010)

Shifted personal correspondence to near-real-time written exchange. Email was more casual than letters but still addressed to a specific person — it retained the intimacy and context of correspondence while lowering the threshold for sending. The volume exploded. Fred’s archive contains ~46,000 messages spanning 2001–2026, with the highest density in 2001–2007.

What survives: Nearly everything sent and received, if exported. No caller ID problem, no memory problem. But the inverse bias of letters applies: because everything was preserved, the signal-to-noise ratio is low. Commercial email, mailing lists, and logistics traffic overwhelm personal content.

The golden window: 2001–2007 is the richest autobiographical period in the email archive. After ~2007, personal communication progressively migrated to Facebook and later to other platforms, leaving email for logistics and professional correspondence. The archive thins not because life got quieter but because the venue changed.

Methods implication: Subject-line triage is essential. Within any month, maybe 15–20% of emails are autobiographically significant. The rest is noise. A second pass after the first triage reliably finds things the first pass missed. See gmail/handoff.md for the working methodology.

3. Social media (~2007–present)

Facebook became Fred’s primary personal broadcast medium from roughly 2007 onward. It serves a different social function than either letters or email: it addresses everyone simultaneously, with no specific recipient. The communication is curated for a mixed audience (close friends, acquaintances, family, colleagues) rather than tailored to one person.

What this means for interpretation:

  • A post is a public performance as much as a communication
  • The emotional register is different — more guarded, more wit, less vulnerability
  • A one-line quote may be the only written evidence of a whole interior period
  • Absence from posting is not absence from living
  • What’s not posted is often as significant as what is

The newspaper test: Social media posts are closer to newspaper columns than to letters. They’re not lies, but they’re edited. The journals and emails from the same period are the unedited version.

What survives: Fred’s Facebook archive contains 13,348 posts and 10,899 comments (2007–2026), all synthesized into self-analysis/facebook-notes-YYYY.md. Unlike email, the archive is nearly complete — Facebook preserves everything. The challenge is interpretation, not recovery.

4. AI-mediated communication (~2025–?)

Fred now communicates significantly through and with AI assistants — drafting, analyzing, and in these sessions, narrating his own life to a system that retains context across conversations. This is a new thing. It isn’t broadcast (like social media) or private correspondence (like letters or email); it’s more like thinking out loud to an unusually attentive interlocutor.

For this project specifically: the Claude conversations are themselves a primary source. Observations Fred makes in passing during archive triage (“sorry, late-stage bitterness creeping in”; “I try to keep that in perspective”) are candid in a way that neither social media nor formal correspondence tends to be. They’re worth capturing.

What comes next is genuinely unclear. AI agents communicating with other AI agents on behalf of users is already technically feasible — scheduling, negotiation, information exchange between autonomous systems. Whether this constitutes “communication” in the human sense (and whether it leaves any record worth reading) is an open question. If it does leave records, they will look nothing like letters or emails or Facebook posts. They will be transactional logs — and the autobiography of that period will require yet another analytical method.

For now: The project’s primary sources for post-2007 life are: Facebook posts, journals (Apple Notes from 2013), and email (logistics/professional). The inner life of 2007–2013 is substantially less documented than the 2001–2006 period. This is a genuine archival gap, not a method failure.

Summary: What each medium preserves

Medium What it preserves What it loses Bias
Phone calls Nothing Everything Total loss
Paper letters Considered thought; selected correspondence Casual conversation; unsaved letters Saved = significant
Email Near-complete record of written exchange Phone calls; in-person conversation Noise overwhelms signal
Social media Public persona; broadcast content; timing Interior life; private exchanges Performance bias
AI conversation Candid thinking-out-loud Depends on what’s logged/exported Potentially the most honest register

The observer effect — reading a private archive is not neutral (note, 2026-06-26)

A phrase worth keeping (heard on NPR, on language): “Any time we inscript, we encrypt.” Writing is already encoding — a private journal arrives pre-ciphered in shorthand, initials, omissions, and a register meant for an audience of one. And like Schrödinger’s cat, the sealed entry sits in a kind of superposition until someone opens the box; the act of reading begins its collapse. Observation here is not passive: to read a private journal is to change it — to fix one reading of an ambiguity the writer left open, to pull the private into a context it was encrypted against.

Three consequences for a project like this:

  • Opening one box opens others. Reading your own journal exposes everyone written about in it — without their voice or consent. The author’s encryption layers (an alias like “patient0,” friends-only locks, initials instead of names, a deliberately semi-private “blog” kept off the indexed web) were the writer pre-managing this collapse. The archivist is the one breaking that seal, decades later, deciding which of those people now gets a visibility flag. Build the public/login/admin model knowing it’s your seal replacing theirs.

  • Reading the dead is a kind of finishing. The effect is sharpest with a deceased person’s private papers (e.g. letters marked “don’t show to X”). There’s no setting the lid back down; reading is also, quietly, an ending. Let the discomfort be information, not something to suppress.

  • Sealed is not preserved — so the real choice is which destruction. The counterweight, and the reason to open the boxes anyway: the unopened archive doesn’t stay pristine. It rots — into dead formats (MacWrite II, WDBNMSWD), into 404’d cloud stubs, into silence. Sealed is just a slower kind of gone. The archivist never actually chooses destroy vs. keep pristine; they choose between the slow destruction (format death) and the fast one (cracking the seal so the words outlive the medium). Opening is a controlled demolition that trades the seal for the content.

For regime 4 (AI-mediated), the observer is inside the experiment: these sessions are themselves primary sources, so analyzing the archive is also adding to it.

The following are challenges that anyone doing a similar project should anticipate:

Format obsolescence. Digital files from the 1980s–1990s are often in unrecoverable formats (Word for Mac WDBNMSWD, etc.). The window for recovering them with classic Mac emulators (Basilisk II, etc.) is closing as the software and hardware become harder to run. Prioritize these early.

Platform fragmentation. Journals across 30 years span Newton exports, Word for Mac, Apple Notes, and probably other platforms. Each requires different extraction methods. Build a comprehensive inventory before assuming you know what you have.

Photo identification. AI can read image content but cannot reliably identify people by face. Any photo archive requires human annotation. Plan for iterative review sessions rather than batch processing.

Public/private split. Social media (Facebook, Twitter) captures the public persona, not the inner life. The contrast between what’s posted and what’s journaled is analytically valuable — but you have to hold both simultaneously. Neither source alone is complete.

Citation provenance. Claiming something happened is easy; citing exactly where you found it becomes crucial in autobiography. Establish a citation convention early — it is much harder to retrofit. See self-analysis/citation-conventions.md for this project’s convention.

Oral history window. Living family members and friends have knowledge that exists nowhere in the written record. This window closes. Fred’s project has Elise, Amy, Kathie, and various friends who haven’t been formally interviewed. Prioritize oral history before the record closes.

Known Gaps and Limitations

  • 1992–1998 digital originals: Most unrecoverable without Classic Mac emulation
  • Physical journals 1991–1994: In storage, not digitized
  • Email corpus: 46,000+ messages extracted, zero processed
  • Photos: No systematic approach yet; requires Fred’s annotation for people identification
  • Oral history: Living family members (Kathie, Amy, Elise) not yet formally interviewed
  • Gram’s world-trip journal (1962): Kathie has it; not yet accessed
  • Gramp’s unread papers: Canis Dirus series, Condensed Smilodon, others on SSD

For methods specific to family history/genealogy research, see reference/family-history-sources.md.
For autobiography structure, see reference/chapter-concepts.md.