PUPILA: An A-Eye on every job board
Table of contents
Getting started — What · Key features · Onboarding · Forking & personalizing · How it works
Using it — Application tracking · AI Apply · AI per-job review · MCP server
Under the hood — Stack · Architecture · Sources · Pipeline stages · Tuning filters
Reference — Repo layout · Run locally · GitHub Actions · Customization · Security & quality gates · Known upstream issues · Conventional commits · License
What
PUPILA is a local-first job aggregator. Choose for schedule a task or trigger a manual refetch and it pulls from many public sources (adding more is an easy job), filters out the noise (junior roles, non-engineering, onsite-only, etc.), and scores each remaining posting against your CV-derived profile. You wake up to a single sorted table of roles that actually fit, every row links to the original posting, so you can apply directly. That's the core value: getting the data, filtering it, and scoring it. The AI layers below are optional.
Optional layers, powered by your local LLM CLI (claude / codex / gemini / opencode — no API keys, no cloud, no third party ever sees your data):
- AI per-job review — a
strong-match/match/weak-match/skipverdict with a one-line reason next to each title. - Jinder — a Tinder-style swipe deck for triaging the top matches in seconds: right-swipe to queue a role for application, left-swipe to skip it forever.
- AI Apply — drafts a tailored cover letter + highlights + Q&A package from your CV and the posting for every job in the queue, while you do something else.
- MCP server — exposes everything the UI can do (17 tools: query / detail / mark applied / enqueue / queue status / aggregator runs / profile regen) to any MCP client so you can drive Pupila from inside Claude Code, Claude Desktop, or Cursor. One-command install. Jump to section ↓
Key features
The stuff that makes the daily routine actually pleasant:
- Day-over-day diff. Every run computes what's new and what disappeared since yesterday — surfaced as ✨ New since last run + 🗑 Removed since last run at the top of
JOBS.md. The actionable bit is always the first thing you see. - Scheduled, no babysitting.
scripts/install-launchd.sh(macOS) andscripts/install-cron.sh(Linux) install two agents — aggregator + AI review — that run on independent times. launchd catches up missed runs after wake. - Group-by-company (default on). Folds a 24-role Vercel into one expandable row instead of dominating the top of your table.
- URL-encoded view state. Every filter, sort, and expanded row syncs to
?q=...&cat=...&src=...— bookmark a filtered view and it rehydrates next time. - Source-health banner. A 🚨 banner appears in
JOBS.mdwhen a fetcher returns zero items or errors, so silent upstream breakage isn't silent. - Application tracking. Click a status pill on any row (
📝 applied / 💬 interview / 🎯 offer / ❌ rejected / ⏸ withdrawn) — saves toconfig/applied.jsonand feeds the "📋 Application status" section at the top ofJOBS.md. - 9-panel Settings dashboard: switch LLM CLI / install or remove the scheduler / regenerate scoring profile / manage job sources (incl. LLM-powered company discovery) / inspect the last run / check disk usage / clean / view environment / monitor the apply queue, all from
pnpm run ui→ Settings. - Talk to your data from an AI client. Optional MCP server exposes every actionable UI surface (filter jobs, mark applied, enqueue AI Apply, trigger a refresh, regenerate the scoring profile) to Claude Code / Claude Desktop / Cursor via 17 typed tools. One command (
bash scripts/install-mcp.sh) wires it in. - Reach past the curated boards. The
bluedoorsource taps a free aggregator over ~1.6M postings across 31 ATS providers (Workday, iCIMS, Oracle, ADP, …) you can't query directly — pulled by your accepted regions, with companies you already follow auto-skipped so it only adds long-tail listings. - RSS feed.
data/feed.xmlgets every "✨ new" job — point any RSS reader at the localfile://path. - Local-first by design. No API keys, no cloud, no hosted scheduler. The LLM features use your existing
claude/codex/gemini/opencodesubscription via the CLI. Your CV, brief, and applied list never leave the machine.
Onboarding
The first time you run pnpm run ui, a three-step wizard sets you up. Total time: about 30 seconds.
1. Pick your LLM CLI
PUPILA shells out to whichever AI CLI you already have authenticated locally — claude, codex, gemini, or opencode. The wizard auto-detects what's on your PATH and pre-selects the first available one. No API keys, no signup, no per-token billing — it uses your existing CLI subscription.
2. Drop your CV
Drag a .pdf / .docx / .md / .txt CV onto the drop zone (or click Choose file). The CV is parsed locally and sent to your LLM CLI to generate a short candidate brief — who you are, what stack you work in, what kind of role you want, and what to avoid. The original CV stays on disk at config/cv.<ext> (gitignored) so AI Apply can re-attach it later.
No recent CV? Click Import from LinkedIn instead (optional) and upload a LinkedIn profile PDF — on your LinkedIn profile, go to More → Save to PDF, then drop the downloaded file here. It runs through the same parse → LLM pipeline with a LinkedIn-tuned prompt, so the resulting brief is comparable to a CV-sourced one. (No login or scraping — the "Save to PDF" export is fully self-serve.)
3. Confirm the generated brief
Eyeball the brief and edit anything that needs tweaking — this natural-language description drives every scoring decision and AI verdict downstream. Hit Looks good and the wizard stamps config/preferences.json with onboardedAt, never re-triggers, and kicks off the first aggregator run automatically. You land on the Jobs view already populated.
How it works
A config-driven, local-first daily job aggregator. Pulls listings from ~20 public sources (job boards, RSS feeds, Hacker News, five ATSes — Greenhouse, Ashby, Lever, Recruitee, Personio — no-key remote feeds like Jobicy and Himalayas, a custom Aave scraper, an Ashby-private GraphQL fetcher, and bluedoor, a free cross-ATS aggregator (~1.6M postings) queried by your accepted regions; adding a new source is one file), normalizes them into a single shape, scores each one against your profile (defined in config/profile.json), deduplicates, and writes the result to your local checkout.
Designed to be cloned and run locally. No hosted scheduler, no public dashboard, no external services. Anyone — frontend, backend, mobile, data, infra, any seniority, any region — sets their role via:
- A short candidate brief generated from their CV (run
pnpm run setup-brief --file ~/cv.pdf, or use the local UI's Profile tab). - Tunable scoring weights and keyword lists in
config/profile.json. - Tier-S company slugs in
config/slugs.json.
Then schedule the aggregator with scripts/install-launchd.sh (macOS) or scripts/install-cron.sh (Linux). Two agents are installed by default: one for the aggregator, one for the AI per-job review.
After onboarding, config/profile.json is auto-generated from your candidate brief by /api/profile-generate (scoring weights + keyword lists tuned to the stack you actually work in, and an exclusion list pulled from your brief's "what to avoid" section). Re-runnable from the Settings tab → Scoring profile → Regenerate.
Contributing rules and project invariants live in CONTRIBUTING.md.
Looking for today's matches? →
JOBS.md(auto-generated by the local aggregator). Raw data lives indata/jobs.json. Local RSS feed atdata/feed.xml(point your reader at the file:// path). Prefer a UI? →pnpm run uiopens a local-only Vite dashboard athttp://127.0.0.1:5173. Four tabs: Jobs (filter, search, sortable columns, click-to-expand rows overdata/jobs.json), Jinder (Tinder-style swipe deck — right-swipe to queue an AI Apply, left-swipe to skip), Profile (drop a PDF/DOCX/MD CV — or import a LinkedIn profile PDF — to set up or refresh your candidate brief), and Settings (scheduler lifecycle, scoring profile regenerate, disk usage, apply queue). See AI Apply and AI per-job review below.
Forking & personalizing
The repo ships with neutral defaults. To make it yours:
1. Clone or fork
gh repo fork FranRom/pupila --clone
cd pupila
pnpm install
Or click "Fork" on GitHub, then git clone <your-fork>.
2. Generate your candidate brief (required)
The brief at config/candidate-brief.md is the natural-language description of who you are, what you want, and what to avoid. This step is mandatory — pnpm run dev will refuse to start until the file exists. (Bypass with PUPILA_NO_BRIEF_CHECK=1 if you genuinely want raw aggregation with no AI review.)
The file is gitignored — it contains CV-derived personal information and should never be committed. setup-brief and the onboarding wizard write to the gitignored canonical path.
The friendliest path is the first-run onboarding wizard: run pnpm run ui on a clean repo and the UI walks you through (1) picking your LLM CLI, (2) dropping your CV, (3) confirming the auto-generated brief. After "Looks good", config/preferences.json is stamped with onboardedAt, the wizard never re-triggers, and you land on the Jobs view.
For the CLI-only path, run setup-brief directly:
pnpm run setup-brief --file ~/Documents/cv.pdf # PDF
pnpm run setup-brief --file ~/Documents/cv.docx # Word document
pnpm run setup-brief --file ~/Documents/cv.md # markdown
pnpm run setup-brief --linkedin ~/Downloads/profile.pdf # LinkedIn "Save to PDF" export
cat resume.txt | pnpm run setup-brief # stdin
--linkedin runs the same pipeline as --file but tells the LLM the input is a LinkedIn profile export (so it ignores LinkedIn's boilerplate). Don't have a recent CV? On your LinkedIn profile, go to More → Save to PDF and pass the downloaded file. (A --file whose name contains "linkedin" is auto-detected as a LinkedIn source.)
Or open the UI and use the Profile tab:
pnpm run ui # http://127.0.0.1:5173 → Profile tab → drop your CV (or "From LinkedIn")
The auto-detected provider order is claude → codex → gemini → opencode (whichever is on PATH first). Override with PUPILA_LLM=codex pnpm run setup-brief .... No API keys; uses your existing CLI subscription.
The two personalization layers, briefly:
config/profile.json(committed defaults) controls what gets fetched + scored (weights, keyword lists, tier-S slugs).config/candidate-brief.md(gitignored, CV-derived) controls the per-job AI verdict (pnpm run ai-review). Both matter for the full daily flow.
3. Scoring profile (auto-generated, hand-tunable)
config/profile.json drives what gets scored. After onboarding finishes, /api/profile-generate runs your local LLM CLI on the brief and fills in:
- Weights: non-zero values for stacks/domains the brief actually mentions (e.g.
stackPrimary: 10if you work with React/Next/TypeScript daily).roleTitle/roleBodyprice a role match. - Keyword lists:
stackPrimary,stackRn,stackOther,w3*,ai*— populated from your stack. roles: the target job titles you're after (e.g. Senior Frontend Engineer and Product Engineer). Each role'stitleMatchtags a posting, earns theroleTitlebonus, and rescues it from the title-based hard drops. Edit them as chips under Profile → Role interests, or let regeneration re-derive them from the brief.titleExcludedSpecialties: pulled from the brief's "what to avoid" paragraph. A frontend brief gets(backend|data|devops|sre|...) engineers?here so those titles hard-drop (unless a declared role rescues them).
Universal hard-drop rules (junior/exec/sales/recruiter/support) and seniority weights stay at sensible defaults regardless of the brief.
Location & work preferences (Profile → Location). This decides which jobs are a geographic fit for you. You set three things:
- Based in — the country you live in (e.g. Spain). This is the only field most people need. Setting it automatically fills in the regions you can work in (Spain → Europe, EMEA, EU, Spain).
- Work types — Remote, Hybrid, On-site. If you don't pick On-site, jobs that are strictly on-site get dropped.
- Accepted regions (advanced, optional) — the regions a job can be tied to and still match you. Auto-derived from your country; only open the Customize panel if your reach is different (e.g. based in Spain but also open to US-remote, or a specific timezone like CET).
How the matching works, in plain terms — a remote job is kept unless it requires a location you can't be in:
| The job says… | Result |
|---|---|
Remote, Worldwide, Global, Anywhere (no specific place) |
Kept — where the company is doesn't matter |
A region you accept (Europe, EMEA, your country) |
Kept |
Remote - US but the description welcomes Europe |
Kept — the label isn't a hard requirement |
Remote - US, Remote (US only), "must be based in / authorized to work in the US" |
Dropped — a real requirement you can't meet |
This is fully neutral — no country is special. A Spain-based profile drops US-only jobs; a US-based profile drops Europe-only jobs; both run the same code. The "Only show jobs in my accepted regions" toggle controls strictness: on = drop out-of-region jobs; off = keep them but rank them lower.
Your regions also drive the
bluedoorsource. Besides filtering, Based in + Accepted regions are the search terms used to query the bluedoor aggregator (one search per region). Widen your accepted regions and bluedoor reaches further; a neutral/empty location simply pulls nothing from it. No geography is hardcoded — the same code works for any country.
Re-run after editing the brief by going to Settings → Scoring profile → Regenerate (or POST /api/profile-generate). To hand-tune, edit config/profile.json directly — your edits won't be overwritten unless you regenerate.
Tweak, run pnpm run dev, inspect JOBS.md, repeat.
4. Update the company slug list
Two ways to follow more (or fewer) Ashby / Greenhouse / Lever companies:
- From the UI (recommended):
pnpm run ui→ Settings → Job sources. Add or remove a company per ATS, Verify a slug against the live board before saving, and run Check board health to flag any boards that 404 or time out. Or click Discover more sources for my profile to let your local LLM CLI propose companies that fit your profile + candidate brief — each suggestion is live-probed against the supported ATSes and ranked by how many of its open roles match you, then you pick which to add. Changes are written toconfig/slugs.local.json(gitignored) as a per-ATS delta on top of the shipped list, so your personal picks stay separate and you keep getting upstream additions toconfig/slugs.json. - By hand: edit
config/slugs.jsondirectly (the committed, shared baseline).
Slugs come from the URL of the careers page (jobs.ashbyhq.com/<slug>, boards.greenhouse.io/<slug>, jobs.lever.co/<slug>). Either way, 404s are silently skipped, so trial-and-error is safe. Adding a slug also auto-excludes that company from bluedoor: the dedicated fetcher is authoritative, so you never get a duplicate from the aggregator.
5. Schedule the daily run
Two agents run on independent schedules so you can tune them separately:
- Aggregator (
pnpm run dev) — fetches sources, scores, writesdata/jobs.json+JOBS.md+data/feed.xml. No LLM needed. - AI per-job review (
pnpm run ai-review) — sends each top-scoring job through your local LLM CLI (claude/codex/gemini/opencode). Skipped via--no-reviewif you don't have a CLI installed.
macOS (launchd, recommended):
./scripts/install-launchd.sh # both, defaults: aggregate 07:00, review 07:15
./scripts/install-launchd.sh --aggregate-time 06:30
./scripts/install-launchd.sh --review-time 09:00
./scripts/install-launchd.sh --no-review # aggregator only (no LLM CLI)
./scripts/install-launchd.sh --uninstall # remove both
launchctl list | grep pupila # check status
launchctl start dev.${USER}.pupila.aggregate # trigger now
launchd's StartCalendarInterval catches up missed runs after wake — if your laptop was asleep at 7am, it runs once the lid opens.
Linux (cron):
./scripts/install-cron.sh # both, defaults
./scripts/install-cron.sh --no-review
./scripts/install-cron.sh --uninstall
crontab -l # inspect
Manual / casual use:
pnpm run daily # = pnpm run dev && pnpm run ai-review
Logs land in data/launchd-{aggregate,review}.{out,err}.log (macOS) or data/cron-{aggregate,review}.log (Linux). Nothing is pushed anywhere — your applied.json, candidate-brief.md, and data/ai-reviews.json stay on your machine. (The check.yml GitHub Action that runs build + tests + lint on PRs is unrelated to scheduling and stays in place.)
6. Personal data + privacy
These files are gitignored and never committed, so a public fork can't leak them:
| File | What it is | First-run source |
|---|---|---|
config/candidate-brief.md |
LLM-generated CV summary | Onboarding wizard / pnpm run setup-brief |
config/cv.{pdf,docx,md,txt} |
Original CV file | Saved by the onboarding wizard / setup-brief so AI Apply can re-attach it |
config/applied.json |
Your application history | UI Profile → status pills |
config/preferences.json |
Your chosen LLM CLI + onboarding-complete stamp | Onboarding wizard |
config/slugs.local.json |
Your personal add/remove of company slugs (overlay on config/slugs.json) |
UI Settings → Job sources |
data/jobs.json |
Daily aggregator output, tuned to your profile | Auto-created by pnpm run dev |
data/ai-reviews.json |
Per-job LLM verdicts | Auto-created by pnpm run ai-review |
data/applications/<job-id>.md |
AI-generated application packages (cover letter, highlights, Q&A) | Auto-created by AI Apply |
data/apply-queue.json |
Pending / running / done / failed AI Apply jobs | Written by the UI + pnpm run apply-worker |
data/swipe-skips.json |
Persistent left-swipe (skip) decisions from Jinder | Written by the UI |
data/feed.xml |
RSS feed of new matches | Auto-created by pnpm run dev |
JOBS.md |
Daily readable matches table | Auto-created by pnpm run dev |
data/archive/*.json |
Monthly snapshots | Auto-created on day 1 of each month |
data/launchd-*.log, data/cron-*.log |
Local schedule logs | Auto-created by the scheduler |
The UI fetches data/jobs.json and data/ai-reviews.json at runtime from /api/jobs and /api/reviews (provided by the Vite middleware) — that's why a fresh clone with no data files still loads cleanly. If you genuinely want git history of any of these (e.g. you're using a private fork as a personal sync mechanism), remove the relevant line from .gitignore and git add the file.
Reset to a clean slate
pnpm run clean # wipe generated outputs, archives, raw caches, schedule logs, queue state
pnpm run clean:onboarding # reset just the onboarding state (preferences + brief + raw CV) — re-triggers wizard
pnpm run clean -- --all # full fresh-clone reset: everything above PLUS brief, applied,
# swipe-skips, generated AI Apply packages, preferences, and uploaded CV
Idempotent — running on an already-clean state prints nothing to clean. After --all the UI re-routes to the onboarding wizard automatically; no hard refresh needed.
Stack
| Layer | Choice |
|---|---|
| Runtime | Node 22 LTS, ESM modules |
| Language | TypeScript 5.9 (NodeNext, strict) |
| Lint + format | Biome 2.4 |
| Package manager | pnpm 11 (minimumReleaseAge: 1d + strictDepBuilds: true for supply-chain hardening) |
| Tests | Vitest 3 — extensive backend + UI suite covering filters, dedup, utils, applied, salary, feed, aave, ashby-private, AI Apply core, apply queue, swipe-skips, profile bootstrap, plus every co-located UI hook (useJobsData, useApplied, useApplyQueue, useSwipeSkips, useUrlSyncedState, useOnboarding) and key components (AppliedBar, SignalChips, QueueBadge). Tiered keyword weighting + salary-aware sort tiebreak have dedicated cases. |
| Pre-commit | simple-git-hooks (runs lint + typecheck on every commit) |
| HTTP | Native fetch with AbortController (30s timeout, 1 retry on 5xx/network) |
| RSS parsing | fast-xml-parser (only runtime dep) |
| HTML scraping | Inline regex parsers (no cheerio/jsdom) |
| Schedule | Local launchd (macOS) / cron (Linux), two agents: aggregator + AI review |
| Output | Files in your local checkout (data/jobs.json, data/feed.xml RSS, JOBS.md, data/archive/<YYYY-MM>.json on month-start) |
| Static analysis | Biome + tsc + build on every PR via check.yml |
Architecture
┌─────────────────────────────────────────────────┐
│ launchd / cron — local agent, daily │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────── src/index.ts ──────────────────┐
│ │
│ ┌──────── Fetchers (Promise.all) ─────────┐ │
│ │ ashby (45 slugs) greenhouse (8 slugs) │ │
│ │ lever (8 slugs) aave (custom) │ │
│ │ ashby-private (1) cryptojobslist │ ──► raw[] per source
│ │ remoteok remotive weworkremotely │ │
│ │ web3career aijobsnet │ │
│ │ hn-hiring hn-jobs │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ write data/raw/<source>-YYYY-MM-DD.json (gitignored)
│ │ │
│ ▼ │
│ normalize (per source) → Job[] │
│ │ │
│ ▼ │
│ filters.applyFilters: hard excludes + scoring │
│ (boilerplate stripped, body truncated to │
│ first 1500 chars before keyword scoring) │
│ │ │
│ ▼ │
│ dedup.dedupe: by URL, then by company+title │
│ │ │
│ ▼ │
│ sort by fitScore desc, postedAt desc, id asc │
│ │ │
│ ▼ │
│ attach config/applied.json status (by URL hash) │
│ │ │
│ ▼ │
│ diff against previous data/jobs.json (✨ new) │
│ │ │
│ ▼ │
│ strip body field, write data/jobs.json, │
│ monthly archive on day-1, render JOBS.md │
│ │
└───────────────────────────────────────────────────┘
│
▼
local files only -> commit manually if wanted
Each fetcher is isolated: it catches its own errors and returns an empty items array plus error details on failure. A 30-second AbortController timeout caps every HTTP call. One source going down can't break the rest of the run.
Where's the auto-commit step? Removed. The pipeline now runs locally and writes to your working tree. If you want git history of your daily snapshots, add a third launchd/cron task that runs
git add data/ JOBS.md && git commit -m "chore: daily run"(or just commit by hand when you want).
Sources
The ATS fetchers (Ashby, Greenhouse, Lever, plus Recruitee and Personio for EU/DACH coverage) carry the bulk of the high-quality signal — each iterates a curated tier-S slug list. The other sources backfill long-tail listings. bluedoor is a different shape: a free aggregator spanning ~1.6M postings across 31 ATS providers (Workday, iCIMS, Oracle, ADP, …) we can't reach directly. It's queried region-by-region from your profile.location (no hardcoded geography), skips companies already covered by a dedicated fetcher, and sits at the lowest dedup priority so curated copies always win. Works fully on the anonymous tier — no signup required. To raise the per-run request budget, optionally export an API key in your shell (export BLUEDOOR_API_KEY=…); it's free via email OTP (no card). The fetcher reads it from the environment and falls back to anonymous when it's unset.
| Source | Type | Endpoint |
|---|---|---|
| ashby | JSON API | api.ashbyhq.com/posting-api/job-board/<slug> × 45 tier-S slugs |
| greenhouse | JSON API | boards-api.greenhouse.io/v1/boards/<slug>/jobs × 8 tier-S slugs |
| aave | HTML scraper (Next.js NEXT_DATA) | aave.com/careers |
| ashby-private | Ashby private GraphQL × N slugs | jobs.ashbyhq.com/api/non-user-graphql (currently 1 slug: chainlink-labs) |
| lever | JSON API | api.lever.co/v0/postings/<slug> × 8 tier-S slugs |
| recruitee | JSON API | <slug>.recruitee.com/api/offers × 4 tier-S slugs (mostly EU scaleups; structured salary + work-type flags) |
| personio | XML feed | <slug>.jobs.personio.de/xml × 4 tier-S slugs (DACH/EU; no salary, URL rebuilt from slug + id) |
| remoteok | JSON API | remoteok.com/api |
| remotive | JSON API | remotive.com/api/remote-jobs?category=software-dev |
| weworkremotely | RSS 2.0 | weworkremotely.com/categories/remote-programming-jobs.rss |
| remoteyeah | RSS 2.0 (custom tags) | remoteyeah.com/rss.xml — global remote-engineering feed |
| jobicy | JSON API | jobicy.com/api/v2/remote-jobs — no-key remote feed; jobGeo region tags + structured salary |
| himalayas | JSON API | himalayas.app/jobs/api — no-key global remote feed (paged 20/req), structured salary |
| cryptojobslist | RSS 2.0 | api.cryptojobslist.com/jobs.rss |
| web3career | HTML scraper | 5 category pages on web3.career |
| aijobsnet | HTML scraper | aijobs.net (global + EU pages) |
| hn-hiring | Algolia API | latest "Ask HN: Who is hiring" thread |
| hn-jobs | Algolia API | hn.algolia.com/api/v1/search_by_date?tags=job |
| bluedoor | JSON API (aggregator, ~1.6M postings / 31 ATS) | api.bluedoor.sh/job-postings/v1/jobs/search — region fan-out from profile.location |
The Ashby tier-S list covers the AI frontier (OpenAI, Cohere, Perplexity, Cursor, ElevenLabs, Modal, LangChain, LangFuse, LlamaIndex, OpenRouter, Pinecone, Supabase, Neon, Clerk, PostHog, Browserbase, Replit, Runway, Notion, Anyscale, BaseTen, Character, Weaviate) plus web3 (Linear, Ramp, Uniswap, Mysten Labs, Paradigm, Polygon Labs, Base, Blockworks, Succinct, Espresso, Phantom, Polymarket, Alchemy, Stacks, Morpho, Magic Eden, LiFi). Greenhouse adds Anthropic, Vercel, Mercury, Coinbase. Lever adds Binance, Ledger, CoinGecko, CoinMarketCap, Safe, Arbitrum Foundation, Celestia, and Mistral (which moved off Ashby to Lever). Custom first-party coverage: Aave via a Next.js __NEXT_DATA__ scraper, and Chainlink Labs via the ashby-private fetcher (Ashby's private GraphQL endpoint; the same fetcher generalized to a slug array, so any future org whose public posting-API is disabled is a one-line config add).
Adding another source is one new file in src/fetchers/, one entry in Source, one normalizer in normalize.ts, and one line in src/index.ts. See CONTRIBUTING.md for the contributor-facing recipe.
Pipeline stages
1. Fetch (src/fetchers/*.ts)
Each module exports fetch<Source>(): Promise<{ items: Raw[]; errors: string[] }> where Raw is source-specific. Errors are caught internally and aggregated, never thrown. Per-slug isolation in the multi-slug ATS fetchers so 404s on individual companies don't cascade. The 30s AbortController timeout in fetchWithTimeout retries once with a 2s backoff on >=500 and network errors; 4xx errors are not retried (they're permanent). Tier-S slug arrays for the multi-slug ATS sources (Ashby, Greenhouse, Lever, Recruitee, Personio) are loaded from config/slugs.json — adding a company is a non-code change.
2. Normalize (src/normalize.ts)
One function per source maps Raw → Job:
interface Job {
id: string; // sha1 of normalized URL
source: Source; // one of the Source literal names
title: string;
company: string | null;
url: string; // http or https only — non-http schemes are filtered out
location: string | null;
remote: boolean; // inferred from text + tags
body: string; // HTML stripped via regex (in-memory only)
tags: string[];
salary: string | null; // surfaced when source provides it (Ashby/Lever/Remotive/web3career/aijobs)
salaryMin: number | null; // parsed annual integer (USD/EUR/etc., minor units stripped)
salaryMax: number | null; // parsed annual integer
salaryCurrency: string | null; // ISO code: 'USD' | 'EUR' | 'GBP' | …
postedAt: string | null; // ISO 8601
fetchedAt: string; // ISO 8601, set at run-start
fitScore: number; // 0-100, populated by filters
categories: string[]; // ids of every matched config category (multi-label); [] = "Other"
_signals?: JobSignals; // per-job scoring breakdown (see below)
applied?: AppliedEntry; // attached when matched against config/applied.json
}
URLs are canonicalized (utm_* stripped, trailing slash normalized) before hashing into id. Any URL that isn't http(s): is rejected at the filter stage as a defense against javascript: / data: / file: payloads from upstream sources.
Note on data/jobs.json: the body field is stripped from the persisted file (it's regenerable from the URL and bloats the artifact ~10×). body is only present in-memory during a run.
3. Filter + score (src/filters.ts)
Hard excludes (drop entirely). The hard-drop chain is a named-rule list (HARD_RULES in src/filters.ts) — every drop is attributed to a rule name, and the per-rule count is surfaced in the JOBS.md "Dropped — hard filters" stat (e.g. (missing_senior_req=812, title_non_eng_role=64, ...)) so you can see which rule is doing the heavy lifting at a glance.
- Title contains
junior|jr|intern|entry-level|associate|graduate|trainee|apprentice. - Title does not contain
senior|sr|staff|principal|lead|head|director|engineer(s)|developer(s)|architect(s). - Body matches a hard US-only / onsite pattern (e.g.
must be located in the United States,onsite only,relocate to San Francisco). - Title or body matches a non-engineering pattern (
marketing|sales|recruiter|...) and the title doesn't contain an engineering keyword. - Title matches a compound non-engineering pattern that contains the word "Engineer" but isn't real engineering:
customer support engineer,sales engineer,solutions engineer,developer relations|advocate|experience,field engineering|operations,business operations,partner(ships) engineer,forward deployed engineer,implementation engineer,gtm,go-to-market. - Title is a non-engineering executive role:
VP,Vice President,CMO,CRO,CFO,COO. - URL scheme is not
http(s):(security gate againstjavascript:/data:/file:URLs from upstream).
Body preparation for scoring. Before regex-matching for keywords, the body is run through preparedScoringBody: it strips known company-boilerplate sections (EEO, privacy notices, accommodations, "About us") and truncates to the first 1500 characters. This prevents false positives from recruiter footers (e.g. "we use Anthropic Claude internally for support tooling" landing a +20 AI signal on a backend role). Hard-drop checks still see the full body so onsite/US-only language at the bottom of a posting is honored.
Soft signals (additive, capped at 100; weights live in config/profile.json):
| Signal | Weight |
|---|---|
| Category — title/body (or body-only) matches a configured category's keywords (e.g. Web3, AI) | category's weight (e.g. +20) |
Stack — body contains react|next.js|typescript (tiered) |
+10 base |
Stack — body contains react native|expo (tiered) |
+5 base |
Stack — body contains graphql|tailwind|vite (tiered) |
+5 base |
Lead title — title contains lead|staff|principal|head |
+15 |
Senior title — title contains senior|sr |
+10 |
Frontend title — title contains frontend|front-end|fullstack|full-stack|web|mobile |
+10 |
| Frontend body — body contains role-specific frontend phrases (design system, ship components, accessibility, etc.) (tiered) | +10 base |
Location — job matches an accepted region / is remote (driven by the location profile block) |
+10 |
Freshness — postedAt within 7 days |
+10 |
Freshness — postedAt within 14 days (and not within 7) |
+5 |
| Penalty — job region-locked outside accepted regions (when not hard-excluding) | -10 |
Tiered keyword weighting. The "stack/frontend body" signals (rows marked tiered above) count occurrences instead of doing a binary match: 1 mention earns half-weight, 2–3 the listed base weight, and 4+ a 1.5× boost. This lets a posting that mentions "react" eight times in concrete role context outscore one that drops it once in a "nice to have" footer. Category, title, location, and freshness signals stay binary because they're inherently low-cardinality and cheating them with repetition isn't a real concern.
Drop anything with fitScore < 30 after the cap.
Categories are config-driven, not hardcoded. They're defined in config/profile.json#categories (generated from your brief on Regenerate, or edited on the Profile tab). A job is tagged with every category whose keywords match (multi-label), and a category's optional weight also boosts the score. Jobs matching no category render under a synthetic "Other" bucket. So a frontend+web3+AI candidate gets web3 / ai buckets, while a fintech candidate's brief yields payments / compliance instead, with no code change. See the pupila-filters skill for the CategoryDef shape.
Unlike the regex-based keyword lists elsewhere in the profile, category keywords are plain literal terms (web3, smart contract, node.js): what you type is what's stored and shown. They're matched whole-word and case-insensitively by compileCategoryKeywords (in src/filters.ts), which anchors with alphanumeric lookarounds instead of \b, so punctuation-edged terms work (c++, c#, .net) and a dot between words is optional (node.js also matches nodejs). For other variants (agent / agents) list both. Regenerate emits literals too, so the Profile-tab chips never show regex noise.
Auditability — _signals. Every kept job in data/jobs.json carries a _signals object recording which scorin
No comments yet
Be the first to share your take.