How is SkyAlertsApp architected across ingest, enrichment, Neon Postgres, and Vercel?

Answer

Born from a midnight Pluto documentary just 18 hours ago, SkyAlertsApp turns live Vera C. Rubin (LSST) and ZTF telescope alerts from ALeRCE into a personal sky observatory at sky.henseler.app.1

  1. At 12:00 AM on 2026-11-01, watching Destination Pluto: Beyond the Flyby sparked a question about ML in the Kuiper Belt and open telescope APIs; by 12:31 AM, SPEC.md was written.1
  2. Every night, Rubin's 3,200-megapixel camera subtracts reference sky templates from fresh exposures to emit alerts whenever anything in the sky brightens, dims, or moves.2
  3. Two isolated Vercel projects and least-privilege Neon Postgres 18 roles (skyalerts_ingest, skyalerts_ro, neondb_owner) ingest those alerts safely at 1 req/s.3
  4. Phase 2 enrichment pulls light curves (up to 200 detections) and top-3 ML probabilities for the 40 highest-priority cosmic events each day inside a 240s budget.4

Applies toSkyAlertsAppOrigin to Phase 22026-11-01

1

From Pluto at midnight to Rubin LSST: what is an astronomy alert?

A late-night documentary about New Horizons' 2015 Pluto flyby led directly to tapping the largest automated sky survey in human history.2

12:00 AM (Couch)
Watching Destination Pluto: Beyond the Flyby on Curiosity Stream sparked questions about New Horizons photography and ML for Kuiper Belt Objects.
12:15 AM (Discovery)
Explored with Claude whether any observatory offers a public API for live sky data, uncovering the Vera C. Rubin Observatory and the ALeRCE broker.
12:31 AM (SPEC.md)
Locked in the 4-phase architecture (SPEC.md): daily Python ingest, Neon Postgres, Next.js dashboard at sky.henseler.app, Resend digest, and podcast export.
Day 1 Build (18 Hours)
Shipped Phase 0 live ALeRCE exploration, Phase 1 database migrations and Vercel cron ingest, and Phase 2 light-curve enrichment and web dashboard.
Vera C. Rubin (LSST)
An 8.4-meter ground telescope on Cerro Pachón, Chile with a 3,200-megapixel camera (40 full moons per shot) scanning the southern sky in 6 bands (u, g, r, i, z, y).
ZTF (Palomar)
The 48-inch Zwicky Transient Facility in California scanning the northern sky in 3 bands (g, r, i), paired with Rubin as our secondary stream.

How do astronomers spot one exploding star or moving Kuiper Belt rock among billions of static stars? They use difference imaging (template subtraction) and stream the residuals to ML brokers like ALeRCE.5

Alert Pipeline ConceptWhat Happens at the Telescope & BrokerHow It Appears in SkyAlertsApp Data
1. Science ExposureRubin takes a fresh 30-second image of a sky patch through one filter (u, g, r, i, z, y).Stored as scienceFlux and scienceFluxErr (in nanoJanskys, nJy) inside detections.raw.
2. Template SubtractionThe pipeline aligns and subtracts a deep reference map (templateFlux) of that exact sky coordinate.Static stars cancel to zero; any leftover light becomes a detection (measurement_id / candid) with difference flux psfFlux.
3. Positive vs. Negative FluxIf an object brightened or entered the frame, difference flux is positive; if a variable star dimmed or moved away, difference flux is negative.Physics preservednormalize_detection stores negative flux_njy faithfully and computes magnitude mag = -2.5*log10(psfFlux) + 31.4 only when psfFlux > 0.
4. Stamp Classifier (First Alert)Within minutes of the first detection, a CNN inspects three tiny image cutouts (science, template, difference) to guess the broad class.stamp_classifier and stamp_classifier_rubin_beta_20260421 emit early probabilities for SN, AGN, VS, asteroid, or bogus.
5. Light-Curve Classifier (Multi-Night)Once detections accumulate across nights into a brightness-over-time graph (light curve), a random forest classifies the exact subtype.High precisionlc_classifier refines the class into SNIa, SNII, RRL, CEP, or LPV and takes precedence as objects.top_class.
Priority & CategoryCodebase Class LabelsWhat You Are Actually Looking At in the Night Sky
Tier 1 — Explosive TransientsSN, SNIa, SNIbc, SNII, SLSN, TDE, KN, CV/NovaEnriched firstCataclysmic events: White-dwarf thermonuclear supernovae (SNIa), massive core-collapse supernovae (SNII, SNIbc, SLSN), neutron-star mergers (KN), black holes shredding stars (TDE), and accreting white-dwarf novae (CV/Nova).
Tier 2 — Pulsating & Eclipsing StarsVS, RRL, CEP, LPV, E, DSCT, YSO, MicrolensingDynamic stars in our galaxy: Cosmic distance-ladder pulsators (RRL RR Lyrae, CEP Cepheids), red giants (LPV), eclipsing binary pairs (E), young protostars (YSO), and gravitational lens brightenings (Microlensing).
Tier 3 — Supermassive Black HolesAGN, QSO, BlazarActive Galactic Nuclei: Supermassive black holes at the centers of distant galaxies flickering as they feed on surrounding gas disks (AGN, QSO) or aim relativistic jets toward Earth (Blazar).
Filtered — Solar System & Artifactsasteroid, bogusSkipped by enrichMoving rocks & camera ghosts: Main-belt/near-Earth asteroids and Kuiper Belt cousins move across sky coordinates between exposures (asteroid), while cosmic-ray hits and subtraction ghosts (bogus, 52% of raw LSST alerts) are excluded from enrichment.
2

How do the services, security boundaries, and database roles connect?

SkyAlertsApp end-to-end topology
Diagram structure
  • Vercel Cron 13:00 UTC cron
  • skyalerts-ingest WSGI wsgi
  • ALeRCE LSST API lsst_api
  • ALeRCE ZTF API ztf_api
  • Neon Postgres 18 neon
  • GitHub Actions Migrate gha
  • Next.js sky.henseler.app web
  • GitHub OAuth Admin oauth
  • Resend Email Digest resend
  • cron → wsgi · Bearer secret
  • wsgi → lsst_api · 1 req/s HTTPS
  • wsgi → ztf_api · 1 req/s HTTPS
  • wsgi → neon · skyalerts_ingest
  • gha → neon · neondb_owner
  • web → neon · skyalerts_ro
  • oauth → web · ID 16056018
  • web → resend · Weekly digest

The platform splits ingestion, migrations, and public serving into separate runtimes so no component holds excess database privileges.3

  1. Vercel Cron 13:00 UTC

    Invokes GET /api/cron/ingest once daily with Authorization: Bearer $CRON_SECRET verified via hmac.compare_digest.

  2. skyalerts-ingest WSGI

    Runs inside a dedicated Vercel project (maxDuration = 300s, 240s budget) and throttles ALeRCE calls to 1 req/s.

  3. Neon Postgres 18

    Stores objects, classifications, detections, and run logs in neondb (Postgres 18, us-east-1).

  4. GitHub Actions Migrate

    Applies checksummed db/migrations/*.sql files via direct endpoint behind a production approval gate.

  5. Next.js sky.henseler.app

    Queries Neon through skyalerts_ro (10s timeout, read-only transactions) and never calls ALeRCE on page load.

  6. GitHub OAuth Admin

    Restricts /admin routes server-side to owner GitHub user ID 16056018 (justadakaje) and writes to audit_log.

Role / PrincipalRuntime ConsumerAllowed OperationsPostgres Guardrails
neondb_ownerdb/migrate.py in GitHub ActionsOwner onlyDDL migrations and schema_migrations bookkeepingDirect non-pooler connection; session advisory lock 0x5C7A1E27
skyalerts_ingestskyalerts-ingest Vercel Python functionNo DELETESELECT, INSERT, UPDATE on data tables; column-level INSERT/UPDATE on ingest_runsNOSUPERUSER; statement_timeout = 60s; REVOKE TEMPORARY
skyalerts_roPublic Next.js app (sky.henseler.app)Read-onlySELECT on objects, classifications, detections, ingest_runsdefault_transaction_read_only = on; statement_timeout = 10s
3

How does the daily ingest and Phase 2 enrichment pipeline execute?

Daily ingest and bounded enrichment flow
Diagram structure
  • Claim xact lease claim
  • Compute MJD floor floor
  • Page LSST and ZTF page_src
  • List budget hit? status_gate
  • Select 40 candidates pick_cands
  • Fetch dets and probs enrich_obj
  • claim → floor · lock ok
  • floor → page_src · 2d to 7d
  • page_src → status_gate · upsert batch
  • pick_cands → enrich_obj · Tier 1..3
  • status_gate → enrich_obj · skip if yes
  • status_gate → pick_cands · no

Each run claims a database lease, pages three ALeRCE sources newest-first, computes run status from the list phase, and enriches up to 40 objects.4

  1. Claim xact lease

    Acquires pg_advisory_xact_lock(0x5C7A1E28), marks running rows older than 600s as failed, and inserts a new run row.

  2. Compute MJD floor

    Uses last_success_mjd_floor() - 1.0 day of slack, clamped between 2.0 and 7.0 days before now_mjd.

  3. Page LSST and ZTF

    Pages lsst:default, ztf:stamp_classifier, then ztf:lc_classifier (page_size = 100, up to 20 pages each).

  4. List budget hit?

    Locks in run status (success, partial, or failed) from list paging alone so enrichment never blocks catch-up advancement.

  5. Select 40 candidates

    Queries enrichment_candidates(run_id, 40) excluding bogus and asteroid, ordered by class tier, probability, and lastmjd.

  6. Fetch dets and probs

    Calls detections() and probabilities(), keeps 200 newest detections and top-3 ranks per classifier, and commits atomically.

0100200List Paging (22reqs)Enrichment (40objs / 80 reqs)Headroom BufferSeconds2285133Seconds
Cron 240s time budget allocation across phases (seconds)
4

How do Rubin (LSST) and ZTF payloads differ in live ALeRCE responses?

Live exploration on 2026-11-01 confirmed that LSST and ZTF use separate APIs, incompatible object IDs, and different photometry units.5

DimensionRubin / LSST (api-lsst.alerce.online)ZTF Legacy (api.alerce.online)Normalization Strategy (normalize.py)
Object ID (oid)int64 (e.g. 313708285401235521)str (e.g. ZTF18acwycia)Coerced to printable text (1..64 chars) under composite key (survey, oid).
List ClassificationSingle passIncluded on every row (class_name, classifier_name, classifier_version)Two passesnull unless filtered by classifier query paramPage ZTF twice (stamp_classifier then lc_classifier); lc_classifier wins top_class.
Detection IDmeasurement_id (int64 coerced to text)Doc mismatch fixedcandid (str); no measurement_id field existsMapped to detections.measurement_id; rows missing ID or mjd are rejected.
PhotometryDifference psfFlux and psfFluxErr in nJy (can be negative)magpsf and sigmapsf in magnitudes (0 < mag < 40)LSST stores flux_njy always and computes mag = -2.5*log10(flux) + 31.4 only when psfFlux > 0.
Band MappingInteger band + per-row band_map (6:u, 1:g, 2:r, 3:i, 4:z, 5:y)Integer fid (1:g, 2:r, 3:i)Resolved to canonical band text (u, g, r, i, z, y) with fixed fallback map.
Probabilitiesclassifier_version is an integer (1, 201, 202)Up to 338 rows across 26 (classifier, version) pairsVersion coerced to text; keep ranking <= 3 per (classifier_name, classifier_version).
Slim raw WhitelistSlim JSONB7 keys (band, psfFlux_flag, isNegative, snr, scienceFlux, scienceFluxErr, has_stamp)Slim JSONB10 keys (candid, fid, isdiffpos, rb, drb, dubious, diffmaglim, magpsf_corr, sigmapsf_corr, corrected)Strips 60+ bulky LSST pixel/covariance fields per detection to protect Neon free-tier storage.
Live FreshnessPrimary streamNewest detection ~23 h old (nightly batch)Secondary streamNewest detection 65.5 h old (patchy cadence)Daily cron at 13:00 UTC captures the completed Chilean night after broker processing.
52%48%bogus (filtered from enrichment) · 13 (52%)VS / AGN / asteroid (retained in objects) · 12 (48%)
Live LSST 25-object exploration sample by top class (2026-11-01)
5

How is the Postgres 18 schema structured across migrations 0001–0003?

All tables key astronomical sources by (survey, oid) and track enrichment progress with enriched_at and enriched_lastmjd.6

SkyAlertsApp Postgres 18 schema (migrations 0001–0003)
Diagram structure
  • ingest_runs ingest_runs
  • audit_log audit_log
  • objects objects
  • classifications classifications
  • detections detections
  • objects → ingest_runs · last_ingest_run_id
  • classifications → objects · (survey, oid)
  • detections → objects · (survey, oid)
Per-object enrichment and atomic transaction sequence
Diagram structure
  • ingest.run() runner
  • AlerceClient http
  • normalize.py norm
  • Store (Neon PG) db
  • runner → db · enrichment_candidates(run_id, 40)
  • db → runner · [(survey, oid), ...] ordered by tier
  • runner → http · detections(survey, oid)
  • http → runner · raw detection dicts
  • runner → http · probabilities(survey, oid)
  • http → runner · raw probability dicts
  • runner → norm · normalize_detection + normalize_probability
  • norm → runner · <=200 newest dets + top-3 probs/classifier
  • runner → db · enrich_object(survey, oid, dets, probs)
  • db → runner · COMMIT dets + probs + enriched_lastmjd
6

What is the project roadmap and verification checklist?

SkyAlertsApp progresses through five phases gated by automated tests, CodeQL, dependency audits, and least-privilege role checks.1

PhaseScope & DeliverablesCurrent Status
Phase 0 — FoundationRepo justadakaje/SkyAlertsApp, MIT license, uv lockfiles, CI (ruff, pytest, pip-audit, CodeQL), live ALeRCE explore workflow, docs/alerce-notes.md.Live verifiedCompleted
Phase 1 — Ingest & DatabaseNeon Postgres 18 (neondb), db/migrate.py, migrations 0001 & 0002, defensive AlerceClient, daily cron skyalerts-ingest, lease locking (#17).DeployedCompleted
Phase 2 — Dashboard v1 & EnrichmentMigration 0003_enrichment.sql, bounded light-curve & probability enrichment (#18), Next.js public dashboard + /admin OAuth console at sky.henseler.app.Active sprintIn progress
Phase 3 — Weekly DigestWeekly selector for rare high-confidence transients, plain-English explanations, Resend delivery from sky.henseler.app (us-east-1).Planned (DNS ready)
Phase 4 — Trends & PodcastHistorical class-mix and rate charts, plus weekly Markdown script export for the "what the sky did this week" podcast segment.Planned

Opening diagram…