taskbash

taskbash is a personal chief-of-staff app. Every morning at 7:00 AM Pacific it reads my Gmail, Google Calendar, Granola meeting notes, Linear and Slack. It pulls out the tasks that need me, removes duplicates and anything I already handled, and gives me one ranked list. It also drafts email replies in my voice, writes prep notes before meetings, and sends a summary and meeting reminders to WhatsApp.

I built it alone, starting May 12, 2026, and I use it every workday. This page covers how it works, how I test the AI inside it, and what changed release by release.

How it works

One pipeline runs every morning and whenever I press Re-run. Gmail is also checked every 5 minutes, so new email turns into tasks during the day.

1. Collect. Sources connect through Google sign-in, Nango, Composio and API keys. If a source is down, the run continues without it.
2. Extract. Claude reads each email thread and meeting and lists candidate tasks. Sources run in parallel.
3. Judge. A second model checks every candidate against the source and my open tasks. It keeps, drops, merges or nests each one.
4. Classify. Each task gets work-area tags. The classifier reads preferences learned from my past corrections.
5. Deduplicate and store. Each task is matched against stored tasks in Postgres. It is new, carried over, or suppressed because I already cleared it.
6. Deliver. The ranked list on /today, prep notes with attendee research, reply drafts saved in Gmail, and WhatsApp messages.

Design decisions

• Stored state, not a fresh list. Every run compares new tasks with stored ones. A task I cleared yesterday never comes back.
• Only I close tasks. An early version closed tasks when they disappeared from a source. It closed real work, so I turned it off on May 30, 2026 and restored the lost tasks.
• One wrapper for every model call. Each call is logged with tokens, cost and latency, and traced in Langfuse. Each task links back to the call that made it, which is what made the evaluation below possible.
• A second model reviews the first. The judge sits behind a feature flag, so I can switch it off in production without a deploy.
• Health checks in code. An audit script runs 8 checks against the live database, such as "every new task links to the model call that made it" and "no run is stuck."

How I evaluate it

My eval reported 0% for four months. The model was mostly fine. The test was asking the wrong question.

The setup

When taskbash shows me a wrong task, I mark it as "slop" and pick a reason. I have marked 257 tasks this way. The top reasons were duplicates (139 of 257) and old tasks with past dates (48 of 257).

Some slop marks become eval cases. The original email or meeting is saved, and a job reruns the current extraction prompt on it every 3 days. From June 16 to October 16, 2026, that job ran 86 times on 15 cases, for 599 checks.

What went wrong

0 of 599 checks passed. There were three causes.

1. The wrong question. Each check required the email or meeting to produce no tasks at all. A real email usually holds other, legitimate tasks, so the check could almost never pass. Made-up example: I reject "Join the video call at 12:15" from an email that also asks me to send a proposal. A correct extraction still returns "Send the proposal," and the check called that a failure.
2. A format mismatch. 180 of the 599 outputs did return zero tasks. They still failed, because the model returns {"items": []} and the check only accepted [] or {}.
3. A silent alarm. The job alerts only when the pass rate drops more than 5 points from the last run. A rate stuck at 0% never drops, so it never fired.

The fix

I re-scored every stored output with the question the eval was meant to ask: did the extractor recreate the specific task I rejected? Titles are compared after removing filler words and folding verb variants, so "send the proposal" and "draft the proposal" count as the same task. The script reads the 599 outputs already saved, so the re-score needed no new model calls and cost nothing.

Results

What this tells me:

• The July 7 prompt overhaul worked. The same 8 cases went from 44% to 72%.
• After the overhaul, single runs on the same 8 cases still swing between 62% and 88%, because the model does not answer the same way every time. I judge prompt changes on averages across runs, never on one run.
• Meeting extraction is the weak spot at 52%. Most misses are logistics ("join the call") and old tasks with dates in the past.

Next steps

• Put the corrected check into the eval job, so the dashboard shows the real number.
• Add a flatline alarm that fires when a dataset scores 0% three runs in a row.
• Test duplicate cases at the dedupe step, where duplicates are actually caught, instead of at extraction.
• Grow the test set. There have been no new slop marks since July, so it is stuck at 15 cases.

Changelog

October 11 to 12, 2026

The reply drafter now rates its own confidence. Linear pulls in high-priority, due-soon and in-review issues assigned to me. The Slack step skips cleanly when it is not set up. Fixed a broken check in the audit script.

July 31, 2026

Fixed morning runs timing out on Gmail and meetings (limit raised from 90 to 240 seconds). Cleared tasks can be restored. The Re-run button now runs the real pipeline.

July 6 to 7, 2026

Prompt overhaul: stricter email rules, meeting extraction that reads the transcript first, and suppression of tasks I already cleared. Judge version 2. Every model call moved to Claude Opus 4.7, and a temperature bug in the judge was fixed.

July 1, 2026

Added the judge, a second model that reviews every extractor's output. Tighter duplicate matching.

June 10 to 14, 2026

Built during the BuilderShip hackathon: Slack as a source through Composio, attendee research for meeting prep through Tavily, an optional Llama 3.3 70B classifier on Nebius, learned preferences through mem0, a public landing page, and a live view of each run's steps.

June 9 to 10, 2026

Five fixes from an audit that found about 22% of my open list was slop (39 of 173 tasks): Linear shows only issues assigned to me, meeting extraction skips tasks owned by someone else, calendar logistics are filtered out, duplicate matching folds verb variants, and the classifier tags more generously. Every new task now links to the model call that made it.

Earlier in June: WhatsApp morning summary and meeting reminders through Twilio, error tracking in Sentry, inline editing, and a full QA pass.

May 27 to 31, 2026

Added the slop button, priorities and one-click Gmail send. Every model call logged with cost and latency, plus Langfuse traces. Eval datasets and the 3-day eval job. Reply drafts saved straight into Gmail, written in my voice. Meeting prep that pulls from email, meetings, Linear and the calendar. Cleared tasks stopped coming back.

May 12 to 18, 2026

First version: the /today list on Supabase, Gmail extraction, short briefs for each task, a work-only filter, Google Calendar and Linear, and a connections page.