IOverview
The sermon archive is a self-contained system you can run from your Mac. Audio files live on Amazon S3, served by CloudFront at cdn.cofec.org/sermons/. The transcripts, search index, and editor all live on cofec.org as static files. Nothing on the server runs Python or talks to Deepgram — your Mac does all the heavy work, then uploads plain JSON files.
This manual is organised around what you'll actually do, in the order you'll most likely need to do it. You probably won't transcribe a sermon every week, but when you do, you'll appreciate the step-by-step recipes. Most paths and commands here can be copy-pasted directly into Terminal.
Every Terminal command shown assumes you've activated the project's Python virtual environment (the line beginning source venv/bin/activate). If your prompt doesn't show (venv) at the start, run that line first.
IIAnatomy of the system
Two locations matter: your Mac, where you do the work, and cofec.org, where the result is served.
On your Mac
~/cofec.org/ ├── audio-sermons/ # MP3 files (mirror of the S3 bucket) ├── sermons-prototype/ │ ├── transcribe/ │ │ ├── venv/ # Python virtual environment │ │ ├── transcribe.py # Send audio to Deepgram │ │ ├── scripture.py # Scripture-reference extractor │ │ ├── refresh-scriptures.py │ │ ├── apply-edits.py # Apply contributor corrections │ │ └── output/ # Slim sermon JSONs (working copy) │ ├── site/ │ │ ├── sermons.html # The public page │ │ ├── build-index.py # Rebuild the catalogue │ │ └── sermons-data/ # Deployable JSON files │ └── corrections/ # Inbox for incoming correction emails │ └── applied/ # Audit trail of processed corrections, grouped by date
On cofec.org
cofec.org/ ├── sermons.html # The page itself ├── submit-corrections.php # Receives the contributor form ├── sync.php # Listening-history sync endpoint ├── sync-data/ # Per-user sync JSONs (one per visitor) │ └── .htaccess # Blocks direct .json access ├── lib/PHPMailer/ # Email library (copied from Team Kilimanjaro) └── sermons-data/ ├── sermons.json # Catalogue └── <sermon-id>.json # One per sermon
The MP3 files themselves stay on S3 — they're large, and the existing CDN setup handles them perfectly.
Listening tracking and sync
Visitors who play sermons accumulate a private listening history — which sermons they've sampled, finished, and made notes on. By default this lives in the browser's localStorage. With sync enabled, it also syncs to cofec.org/sync-data/ so visitors can follow their progress across multiple devices.
The architecture is deliberately simple: each visitor is given a long random sync code on first visit (e.g. COFEC-A7B3-K9P2-X4Y8-M2N6). That code is the credential. The browser sends it to sync.php when reading or writing data; sync.php stores the data in a single JSON file at sync-data/<code>.json. No accounts, no passwords, no email, no database. To link a second device, the visitor pastes the first device's code into Settings on the second device, and the two histories merge.
No personally-identifying information is stored — just listening behaviour and the visitor's personal notes. Anyone with a sync code can read that visitor's data, so visitors are advised to keep their code private. The .htaccess in sync-data/ blocks direct file access and directory listing, so the only way to read data is through sync.php (which requires a valid sync code).
IIIFilename conventions
The sermon filenames encode an enormous amount of useful metadata, which the system parses automatically. Understanding the convention lets you (a) recognise what gets extracted, and (b) name future files in a way that produces clean metadata without manual intervention.
The pieces
A typical filename like st-silas-ep-7-7-13-gal-3-16-em.mp3 contains:
| Element | Example | Meaning |
|---|---|---|
| Venue | st-silas | St Silas (currently the only recognised venue) |
| Service | ep | Evening Prayer |
| Date | 7-7-13 | DD-MM-YY → 7 July 2013 |
| Scripture | gal-3-16 | Galatians 3:16 |
| Preacher | em | The Late Rt Rev Edward Malcolm |
Service codes
| Code | Service |
|---|---|
mp or am | Morning Prayer |
ep or pm | Evening Prayer |
hc | Holy Communion |
Preacher codes
| Code | Preacher |
|---|---|
em | The Late Rt Rev Edward Malcolm |
ejm | Rt Rev Edward John Malcolm (his son) |
bf | Rev Brian Felce |
ed | Rev Dr Edgar Dowse |
js | Rev John Shearer |
jh | Rev J Harding |
pr | Rev Peter Ratcliff |
ib | Rev Ian Budgen |
cr | Mr Chris Richards |
gfh | Mr George Hall |
jh2 | Pastor Joseph Hewitt |
pm | Pastor Pooyan Mehrshahi (only when em is absent) |
ds | Rev David Silversides |
is | Rev Istvan Salanki |
tm | Mr Tristan McDonald |
The pm / em disambiguation
em is both a preacher code (Edward Malcolm) and could be confused with an unrelated suffix. pm is both a service code (Evening Prayer) and a preacher code (Pooyan Mehrshahi). The parser uses this rule:
If em appears in the filename, pm means Evening Prayer. Otherwise pm means Pooyan Mehrshahi.
This works because Edward Malcolm's sermons all appear in Evening Prayer slots, and where Pooyan Mehrshahi preached, Edward Malcolm didn't.
Date disambiguation by Sunday
Sometimes a filename is ambiguous about which run of three numbers is the date. For example, st-silas-pm-20-6-6-10-mt-26-40-em.mp3 could be 20 June 2006, 6 June 2010, or something else entirely. The parser uses this rule:
All Morning Prayer, Evening Prayer, and Holy Communion services fall on Sundays. When the parser sees an MP/EP/HC service, it prefers the date interpretation that lands on a Sunday.
In the example above, 20 June 2006 was a Tuesday but 6 June 2010 was a Sunday, so the parser correctly extracts 6 June 2010.
Naming new files
When adding new sermons, use this template:
venue-service-DD-MM-YY-book-chapter-verse-preacher.mp3 # Examples st-silas-mp-12-3-23-rom-8-28-em.mp3 # MP, Romans 8:28, by EM st-silas-hc-19-5-13-gal-3-7-em.mp3 # Holy Communion, Galatians 3:7 st-silas-ep-7-7-13-gal-3-16-em.mp3 # EP, Galatians 3:16 # With verse range: st-silas-ep-2-6-13-gal-3-9-10-em.mp3 # Galatians 3:9-10
Element order is flexible; the parser identifies pieces by their pattern, not their position. But the template above is the cleanest convention going forward.
If a filename can't be parsed
Sermons whose filenames don't yield a date or service won't break — they just appear in the archive with whatever metadata was extractable. The synopsis (generated separately by Claude Haiku) gives every sermon a meaningful presence in the catalogue regardless of how parseable its filename is.
If you ever spot a sermon with bad metadata, you can either rename the source file and re-run backfill-metadata.py, or hand-edit the slim JSON directly (see section VII).
IVTranscribing new sermons
You have a folder of MP3 files locally, and you want them transcribed and added to the archive. Total time per sermon: about a minute of your attention, 30-60 seconds of Deepgram processing.
Every time you add new sermons, you run these four steps in order. The rest of this section explains each one in detail, but this is the recipe to come back to:
cd /Users/jre/cofec.org/sermons-prototype/transcribe source venv/bin/activate export DEEPGRAM_API_KEY="your-deepgram-key" export ANTHROPIC_API_KEY="your-anthropic-key" # 1. Transcribe new MP3s (skips already-done ones) python3 transcribe.py \ --dir /Users/jre/cofec.org/audio-sermons \ --base-url "https://cdn.cofec.org/sermons/" # 2. Generate synopses for the new sermons (skips already-summarised ones) python3 summarise.py # 3. Build the deployable catalogue cd ../site python3 build-index.py # 4. Upload sermons-data/ to cofec.org/sermons-data/
All four steps are idempotent — you can re-run any of them without harm. Skipping step 2 or 3 is the most common mistake; an upload of sermons-data/ that doesn't reflect your latest transcripts almost always means build-index.py hasn't been run since the new transcripts arrived in output/.
Step 1 — Save the MP3s locally
Download the sermons from the S3 bucket (or via the CDN URL) and put them into:
/Users/jre/cofec.org/audio-sermons
Filenames become the sermon IDs (lowercased, with special characters normalised). So defending-av,-part-1.mp3 becomes the sermon id defending-av-part-1. The cleaner the filename, the cleaner the URL.
Step 2 — Open Terminal and step into the project
The easiest way: type cd (with a trailing space), then drag the transcribe folder from Finder into the Terminal window and press Enter.
cd /Users/jre/cofec.org/sermons-prototype/transcribe
source venv/bin/activate
export DEEPGRAM_API_KEY="your-key-here"
From console.deepgram.com. The key only needs to be set once per Terminal session — if you close Terminal and come back tomorrow, run the export line again.
Step 3 — Run the transcription
For a single sermon you've just downloaded:
python3 transcribe.py \ --file "/Users/jre/cofec.org/audio-sermons/SERMON-FILENAME.mp3" \ --title "Human-readable title here" \ --base-url "https://cdn.cofec.org/sermons/"
Or to transcribe everything in the audio-sermons folder in one go (the script automatically skips sermons it's already done):
python3 transcribe.py \ --dir "/Users/jre/cofec.org/audio-sermons" \ --base-url "https://cdn.cofec.org/sermons/"
You'll see progress per sermon — "reading… uploading to Deepgram… done in 38s" — and two files appear in output/ for each: the full Deepgram response (kept as an archive) and the slim JSON the player will actually use.
Step 4 — Build the deployable index
Step into the site folder and run the index builder:
cd ../site python3 build-index.py
This rebuilds sermons-data/ with the catalogue (sermons.json) and per-sermon files. Each rebuild gets a fresh built_at timestamp inside the catalogue, which makes browser caches refresh automatically when you redeploy.
Step 5 — Add synopses and metadata
Before deploying, run backfill-metadata.py and summarise.py to enrich the new sermons. See section V.
Step 6 — Upload to cofec.org
See section VIII. For new transcriptions you'll be uploading the whole sermons-data/ folder.
VAdding synopses and metadata
Every sermon in the archive carries two layers of metadata beyond the transcript itself:
- Filename-derived: date, service, primary scripture text, preacher. Extracted by regex — free, instant.
- AI-derived: a 60-80 word synopsis, optional series name, topical tags. Generated by Claude Haiku for about £0.01 per sermon.
Both populate automatically for new transcriptions (the transcribe script calls them). For sermons transcribed before these features existed, you run the two scripts manually.
Step 1 — Filename metadata (instant, free)
cd /Users/jre/cofec.org/sermons-prototype/transcribe source venv/bin/activate python3 backfill-metadata.py
This walks every slim JSON in output/, parses its filename, and writes the extracted metadata back. Sermons whose filenames don't yield much (no clear date, unrecognised pattern) are left as they are — no harm done.
Use python3 backfill-metadata.py --print to dry-run and see what would be extracted before writing.
Step 2 — Synopses and series detection
You need an Anthropic API key from console.anthropic.com. Once obtained:
export ANTHROPIC_API_KEY="your-key-here"
python3 summarise.py
The script processes every sermon missing a synopsis, calls Claude Haiku 4.5 with the transcript, and writes back synopsis, series, and tags. It's safe to interrupt and resume — already-summarised sermons are skipped automatically.
Before running across the whole archive, try python3 summarise.py --limit 5 to process just five sermons. Look at the resulting synopses in output/*.json, decide whether the tone and brevity feel right. If you want adjustments to phrasing, the prompt is at the top of summarise.py and is easy to edit.
Step 3 — Rebuild and deploy
cd ../site python3 build-index.py
Upload the resulting sermons-data/ folder to cofec.org. The archive page now shows each sermon's synopsis, primary text, date, and preacher; series of sermons collapse under single headings.
Cost calibration
Claude Haiku 4.5 charges per million input/output tokens. A typical 25-minute sermon transcript is around 4,000 words ≈ 5,500 tokens of input, plus ~200 tokens of synopsis output. At current rates that works out to roughly 1¢-2¢ per sermon.
| Corpus size | Approximate cost |
|---|---|
| 25 sermons (current) | ~£0.20-0.40 |
| 100 sermons | ~£0.80-1.60 |
| 670 sermons (full archive) | ~£5-10 |
Re-running on a sermon after corrections costs the same. If you wanted to keep synopses in sync after major edits, you could run summarise.py --force to regenerate everything — but for incidental fixes, the original synopsis remains accurate.
VIWhen a correction arrives
A contributor uses the "Improve transcript" button on a sermon page, makes their fixes, and clicks Submit. You receive an email at john@sas.vc with the subject line beginning 📝 Sermon corrections: and a JSON attachment.
The email itself contains the same recipe with copy-paste blocks. This section is the longer-form version, with explanations.
Step 1 — Save the attachment
Download the .json file from the email and put it in the corrections folder. The filename doesn't matter — the script finds it automatically.
/Users/jre/cofec.org/sermons-prototype/corrections/
If you've fallen behind and have several pending corrections, drop all of them into the folder. The script will process them in chronological order in one pass.
Step 2 — Preview everything
Paste this single block into Terminal. The script scans the corrections folder, finds every unprocessed file, and tells you exactly what each one would change. Nothing is written to disk in dry-run mode.
cd /Users/jre/cofec.org/sermons-prototype/transcribe && source venv/bin/activate && python3 apply-edits.py --dry-run
You'll see each change listed — word edits like p12 w47: 'Bergen' → 'Burgon' and metadata changes like preacher: 'Edward Malcolm' → 'Rev John Smith', grouped by file. A summary at the end totals everything across all pending corrections.
Step 3 — Apply and rebuild
If the dry-run looks right, paste this. It applies every correction, moves the processed files into corrections/applied/<today's date>/ for the audit trail, then runs build-index.py to refresh the deployable catalogue:
cd /Users/jre/cofec.org/sermons-prototype/transcribe && source venv/bin/activate && python3 apply-edits.py && cd ../site && python3 build-index.py
The script's output lists exactly which JSONs in sermons-data/ need uploading — always sermons.json plus one per edited sermon.
Step 4 — Upload
Upload the named files to cofec.org/sermons-data/. After uploading, changes should appear immediately thanks to cache-busting on the JSON envelope. If they don't, hard-reload (Cmd+Shift+R).
Applied corrections are moved to corrections/applied/YYYY-MM-DD/ rather than deleted. To see who contributed what and when, just ls that folder. To re-apply a correction (e.g. after restoring an output JSON from backup), move the file back into the parent corrections/ folder and re-run.
If you'd rather apply a single named file (perhaps to test something), pass it as an argument: python3 apply-edits.py /path/to/corrections_xyz.json. In this mode the file is not archived to applied/ — it stays where you pointed at it.
Within seconds of upload, the new version is live. Any visitor opening that sermon — including the contributor — sees the corrected text.
Step 5 — Reply to the contributor
Just hit Reply on the email. The Reply-To header is set to the contributor's address, so your reply goes straight to them, not to the form server.
Mentioning a specific correction they made ("Loved your catch on Nehushtan — that one would have eluded a regex forever") signals you actually reviewed their work, which is exactly the kind of reinforcement that keeps proofreaders proofreading.
VIIEditing the corpus yourself
You have two options for direct edits.
Option A — Use the editor as a contributor would
Open the sermon on cofec.org, click "Improve transcript", make your fixes, click Submit. The email arrives in your inbox; treat it like any other contribution. This is the simplest path and keeps everything auditable.
Option B — Edit the slim JSON directly
For bulk changes or things the editor can't easily do, you can hand-edit the slim JSON in transcribe/output/. The structure is:
{
"id": "defending-av-part-1",
"title": "Defending the AV, Part 1",
"audio_url": "https://cdn.cofec.org/sermons/...",
"duration": 2205.4,
"paragraphs": [
{
"idx": 0,
"text": "Full paragraph text…",
"start": 0.0,
"end": 28.4,
"words": [
{ "w": "Welcome", "s": 0.16, "e": 0.52 },
...
]
},
...
],
"scriptures": [...]
}
You can edit any w, s, or e value, but don't add or remove word objects in the words array. Word slots are how the audio sync stays aligned with the transcript. If you really need to delete a word, set its w to an empty string — the rendering and search will skip it, but the slot stays in place.
After hand-editing, refresh the scripture references and rebuild:
cd /Users/jre/cofec.org/sermons-prototype/transcribe source venv/bin/activate python3 refresh-scriptures.py cd ../site python3 build-index.py
VIIIDeploying changes to cofec.org
The site is entirely static — there's no database, no build pipeline on the server, no caching layer to flush. Uploading the new files is the deployment.
Which files to upload
| Scenario | Upload these |
|---|---|
| New sermon transcribed | sermons.json + new <sermon-id>.json |
| Correction applied to one sermon | sermons.json + that sermon's .json |
| Several sermons changed at once | The whole sermons-data/ folder (simpler than picking) |
Updated sermons.html |
Just sermons.html |
About browser cache
The catalogue (sermons.json) is loaded with a cache-busting query string on every page visit, so it's always fresh. Per-sermon files use the catalogue's built_at timestamp as their cache-buster, so as long as you ran build-index.py before uploading, refreshing the page picks up the new content automatically.
If you ever updated sermons.html itself and it appears to be running an old version, that's a different cache layer — do a hard reload (Cmd+Shift+R) once.
IXTroubleshooting
"command not found: python"
macOS only has python3, not python. Always use python3.
"externally-managed-environment" when running pip
Your Python virtual environment isn't activated. Run source venv/bin/activate in the transcribe/ folder first; your prompt should then show (venv).
Deepgram returns 403 Forbidden when fetching a URL
This is CloudFront's WAF blocking server-to-server fetches as suspected bots. The transcribe script downloads bytes locally and uploads them to Deepgram directly, avoiding this entirely — make sure you're using --file or --dir rather than passing the CDN URL.
"REMOTE_CONTENT_ERROR" from Deepgram
Same issue as above. The script's current default downloads first then uploads, so this shouldn't recur — but if it does, check that transcribe.py contains the transcribe_file() function.
The site loads but says "Couldn't load sermons-data/sermons.json"
Either the upload didn't reach the right folder, or the folder permissions are wrong. Open https://cofec.org/sermons-data/sermons.json directly in a browser — you should see raw JSON. If you see a 404 or an error page, the file isn't where the page expects it.
The contributor's edits aren't showing up after upload
Confirm build-index.py ran after apply-edits.py. The slim JSON in output/ is the source of truth, but sermons-data/ is what gets uploaded — and build-index.py is what copies the changes across. If you skipped this step, the deployable folder still has the pre-correction content.
The page works but the audio player won't play
Open the browser's developer console and look for errors. The most likely cause is that the audio_url inside the sermon JSON points somewhere unreachable. Confirm by clicking the URL directly — if it doesn't play in a fresh tab, the MP3 isn't where the JSON says it is.
Form submission fails — "Email server rejected the message"
Check the SMTP credentials at the top of submit-corrections.php. The mailbox used must exist in SiteGround's email panel and the password must match. The Team Kilimanjaro pattern uses port 465 with ENCRYPTION_SMTPS.
summarise.py fails with "ANTHROPIC_API_KEY not set"
You need to set the API key in the current Terminal session before running. export ANTHROPIC_API_KEY="your-key", then re-run. As with the Deepgram key, the export only lasts for the open Terminal session.
summarise.py returns "JSONDecodeError"
The model occasionally wraps its JSON output in a markdown code fence or adds preamble despite instructions. The script handles common cases, but if it fails on a particular sermon, re-running usually succeeds. If a specific sermon fails consistently, look at the transcript — extremely short transcripts (very short sermons) can confuse the model; you can write the synopsis by hand in output/<id>.json as a workaround.
Filename parser extracts the wrong date
If a Sunday service got a non-Sunday date, the underlying filename is probably mislabelled (a typo in the original recording's filename). The Sunday rule kicks in only when there's ambiguity; an unambiguous wrong date is preserved verbatim. Either fix the filename and re-run, or correct the date directly in the slim JSON.
A sermon appears as part of an incorrect series
Series detection happens two ways: (a) the LLM proposes one during summarise.py, (b) the page falls back to regex-detecting "Part N" patterns in the title. Both can occasionally over-group. If a sermon was wrongly assigned to a series, set the series field to null in output/<id>.json and rebuild — the page will then treat it as standalone.
XFile reference
Scripts you run
| File | Purpose |
|---|---|
transcribe.py | Send audio to Deepgram, produce slim JSONs in output/. Automatically calls the filename parser and scripture extractor for new sermons. |
backfill-metadata.py | Re-parse every sermon's filename to attach date, service, scripture, preacher metadata. Free, no AI. |
summarise.py | Call Claude Haiku to generate synopsis, series detection, and tags. Costs ~1-2¢ per sermon. |
refresh-scriptures.py | Re-run scripture extraction on all existing slim JSONs (no Deepgram calls, free) |
apply-edits.py | Merge a contributor's corrections file into the matching slim JSON |
build-index.py | Rebuild the deployable sermons-data/ from output/ |
Scripts that support, never run directly
| File | Purpose |
|---|---|
scripture.py | The regex engine for finding scripture references in transcripts. Imported by other scripts. |
filename_parser.py | Extracts date, service, scripture, preacher from sermon filenames. Imported by transcribe.py and backfill-metadata.py. |
Files on the server
| File | Purpose |
|---|---|
sermons.html | The public-facing page. Static. |
submit-corrections.php | Receives the contributor form, emails it to you with the JSON attached. |
sync.php | Stores each visitor's listening history and notes. Anonymous, identified by long random codes; per-user JSON files, no database. |
sync-data/ | Directory of per-user listening data. Protected from public listing by .htaccess. |
lib/PHPMailer/ | The PHP email library, copied verbatim from Team Kilimanjaro. |
sermons-data/sermons.json | The catalogue. Loaded once on page open. |
sermons-data/<id>.json | One per sermon. Loaded lazily when a sermon is opened. |
XISetup from scratch
You'll only do this if you're setting up on a new Mac, or rebuilding the environment after some catastrophic event. The rest of the time, skip this section.
Prerequisites
- Python 3 (already on macOS — no install needed)
- A Deepgram account with API key (console.deepgram.com)
- SFTP or web-based file manager access to cofec.org
One-time Python setup
cd /Users/jre/cofec.org/sermons-prototype/transcribe python3 -m venv venv source venv/bin/activate pip install requests
This creates a private Python sandbox inside the transcribe/ folder and installs the one library the script needs. You do this once; from then on you just source venv/bin/activate each session.
Server setup
- Copy
lib/PHPMailer/from teamkilimanjaro.com to cofec.org (any FTP/file-manager method). - Create a mailbox in SiteGround for cofec.org, e.g.
webform@cofec.org. - Open
submit-corrections.phpand fill in the SMTP CONFIG block at the top with your mailbox credentials. - Upload
submit-corrections.phpandsermons.htmlto the cofec.org document root. - Upload
sync.phpto the cofec.org document root. - Create the
sync-data/directory under the document root, set it writeable by the web server (typically755or775permissions), and upload the supplied.htaccessinto it. - Run
build-index.pylocally; upload the resultingsermons-data/folder to cofec.org.
That's the whole server-side. There's nothing to keep running, nothing to schedule, nothing to monitor.