Prompt·Workflows·DBR02
Make ingestion simpler and cheaper
Less work per import, without losing provenance or corrections.
Prompt
Make ingestion simpler and cheaper Less work per import, without losing provenance or corrections. Work through this in order and check in with me after each step. Stay read-only until I approve a change. 1. Start: PG05 when it is unclear who writes what; otherwise PG07. Map every place that writes to the database (PG05) Map the actual write paths for the important tables in this application. Identify every script, job, API, function, trigger and agent that can create, update or delete their records. Show who owns each field, which paths overlap, and where two writers can disagree or overwrite each other. Recommend the simplest ownership model supported by the existing system. Is our ingestion process more complicated than necessary? (PG07) Trace one complete ingestion cycle from the original source to the final database records. Identify repeated parsing, unnecessary intermediate states, duplicate validation, repeated reads, redundant transformations and writes that could be consolidated. Recommend the simplest workflow that preserves accuracy, traceability, failure recovery and the ability to process corrections. 2. If needed: PG06 for writes that change nothing; PG08 for replay or correction failures; PG09 for validation done in several places. Stop rewriting data that hasn't changed (PG06) Investigate how much database write activity actually changes meaningful information. Find repeated upserts, unchanged updates, unnecessary delete-and-reinsert operations, repeated index maintenance and cascading refreshes. Measure their cost where practical and determine which writes can be eliminated without changing correctness, history or recovery behavior. Can we rerun everything without damaging the data? (PG08) Examine the ingestion and correction workflows under retries, interrupted runs, duplicate source files, out-of-order arrivals and repeated execution. Demonstrate where these cases produce duplicate records, lost corrections, partial results or unnecessary rewrites. Find the smallest durable approach that makes each operation safely repeatable. Find where write logic should live (PG09) Trace how records are validated, transformed and written across application code, database functions, triggers and background jobs. Find rules duplicated in different layers or placed where they are difficult to test and maintain. Recommend the simplest ownership boundary for each rule, without assuming that moving more logic into Postgres is automatically better. 3. Change: Make the smallest change to one cause at a time, keeping provenance and corrections. 4. Done when: The same results for reference inputs, measurably less work, and retries that do no harm.
Related prompts
- Map every place that writes to the databasePromptSeveral jobs, functions, APIs or agents change the same data, or nobody is sure who owns it.
- Stop rewriting data that hasn't changedPromptUpserts and updates run at high volume, imports are costly, or vacuum and write-ahead log churn look high.
- Is our ingestion process more complicated than necessary?PromptThe import pipeline is long, slow to correct, or does the same work at several stages.
- Can we rerun everything without damaging the data?PromptImports get retried, schedulers overlap, files are replayed, or runs fail halfway.
- Find where write logic should livePromptBusiness rules are split across SQL, triggers, edge functions, server routes and workers.
More in Workflows.