A deployment pipeline stopped at its migration step. A check of PostgreSQL activity showed no active queries and no sessions idle in transaction. The latest container log showed the immediate cause. drizzle-kit push 0.30.4 had opened an interactive menu inside a one-off migration job, where no operator could answer it.

The command that launched it was:

yes | npx drizzle-kit push

That looked like a non-interactive command. It actually supplied an endless stream of input that the menu did not treat as a submission. Removing push exposed a psql-only command and SQL that could not run twice. The migration ledger that replaced the old loop then introduced a baseline bug that could mark new migrations as already applied.

The migration job blocked the deployment

Earlier that evening I had changed the deployment workflow so a one-off container job ran the database migrations before the application deployment:

migrate:
  needs: build
deploy:
  needs: [build, migrate]

This dependency graph prevents an application release from passing a failed migration. It also means a migration command that never reaches a terminal state can hold the deployment for as long as the process runs.

The migration script had two phases. Phase one ran drizzle-kit push to reconcile the ORM schema. Phase two looped through every hand-written .sql file and ran it on every deployment. The script recorded no migration history. A comment claimed every SQL file was idempotent and safe to repeat. Both phases contained assumptions that the automated path had not tested.

PostgreSQL showed no active query when I checked

The first lines of the container log stopped at the schema-introspection spinner. That looked like a database query that had failed to return, so I checked pg_stat_activity.

The sessions visible at that moment showed no active query, and no session idle in transaction. That evidence made an active database-side wait unlikely, but it did not prove that no lock existed. I had not queried pg_locks, and a session-level advisory lock can remain after a session becomes idle. PostgreSQL uses the idle state to mean that a backend is waiting for another client command.

The useful conclusion was narrower than my original incident note: the connections visible in that check showed no active query or open transaction. I needed the latest container log tail to see what the client was doing.

My first log query had returned the beginning of the container output. It showed introspection but missed the state that followed it. The current tail contained terminal cursor-control sequences and a select menu. In simplified form, the prompt looked like this:

Was parent_table created or renamed from another table?
> create parent_table
  rename child_2026_01 to parent_table
  rename child_2026_02 to parent_table

The process had completed introspection and was waiting for keyboard input.

The incident used drizzle-kit 0.30.4. Its prompt code used Hanji to handle a select widget and the terminal submission separately. This simplified pseudocode shows the two relevant checks:

select.consume = (key) => {
  if (key.name === "down") moveSelection(1)
  if (key.name === "up") moveSelection(-1)
}

terminal.onKey = (key) => {
  if (key.name === "return") submitSelection()
}

The yes command writes y\n repeatedly. In a reproduction of this input path, Node reported the letter as y and the line feed as enter. Hanji ignored y because it did not move the selection. The terminal ignored enter because it checked for return. The stream supplied no recognised navigation key and no return key. The default choice needed only return, but yes never produced it.

For version 0.30.4, I found no documented non-interactive mode that could express the intended answer to this rename prompt safely. Drizzle documents --force as automatic acceptance of data-loss statements. That behaviour makes it unsuitable as a general safety control for an unattended production migration.

What version 0.30.4 saw in the partitioned schema

The database already contained six parent tables partitioned by month. The SQL migration defined fifteen monthly children and one default child for each parent. That design creates 96 partition children.

The PostgreSQL introspection query in drizzle-kit 0.30.4 selected these relation kinds:

WHERE c.relkind IN ('r', 'v', 'm')

PostgreSQL assigns relkind = 'r' to each partition child and marks it separately with relispartition = true. A partitioned parent uses relkind = 'p'. Because 0.30.4 filtered out p and ignored relispartition, its result included the 96 children as tables but omitted the six parents.

The ORM schema still declared the six parents. From the differ's perspective, those declared tables were missing, and the partition children were unrelated ordinary tables. It therefore asked for one decision per missing parent and offered child tables as rename candidates. The migration definition proves the 96-child layout. I did not preserve the complete live diff, and the database also had a runtime-managed table outside the TypeScript schema. I therefore cannot claim that 96 represented every unknown table in that execution.

The prompt selected create table by default. Submitting that choice would have tried to create a parent table that already existed in PostgreSQL but remained invisible to the introspection query. That response would have failed for a different reason. A later Drizzle issue reports the same omission in version 0.31.10 and notes that a 1.0 release candidate includes p in the query. This article describes version 0.30.4 and does not assume the same behaviour in other releases.

An earlier timeout hid the failure

An earlier post-merge hook used:

timeout 30 npx drizzle-kit push --force 2>&1 || true

The timeout bounded the wait, and || true then discarded the failure status. That let the post-merge helper continue without proving that the database schema was ready. Using the same pattern in the migration gate would allow deployment to continue after a timeout or migration error.

A production migration timeout must keep its nonzero exit status, preserve the diagnostic output and leave the deployment blocked. In this case, the safer repair removed push from the production migration job because reviewed SQL files already owned the production schema history.

The SQL runner then exposed two more faults

After I removed push, the job reached the hand-written SQL files. It failed at this line in the partitioning migration:

set ON_ERROR_STOP on

This is a psql client command. The Node migration runner sent the file to PostgreSQL through node-postgres, so the server parser received the backslash command and rejected it before running the SQL below it.

The same file ended with six ALTER TABLE ... ADD CONSTRAINT ... UNIQUE statements. ADD CONSTRAINT has no IF NOT EXISTS form. A later run would fail when it reached a constraint that already existed.

I removed the psql command and wrapped the six constraint operations in one DO block that checks pg_constraint before each one. That repaired the existing file. The larger problem remained: the runner still depended on every historical migration staying safe to repeat forever.

The migration ledger recorded each file once

I added a tracking table that records each migration filename:

CREATE TABLE IF NOT EXISTS schema_migrations (
  filename   text PRIMARY KEY,
  applied_at timestamptz NOT NULL DEFAULT now()
);

The runner starts a transaction for one migration, executes the file, inserts its tracking row and commits both together. If the file fails, the transaction rolls back, and no tracking row remains. Later deployments skip the filenames already present in the ledger. This removes the requirement to run every historical file on every deployment. It also creates a bootstrap question for databases that already contain the schema but predate the ledger.

The baseline required a sentinel and a fixed cutoff

The first tracked production run would find an empty ledger even though the historical migrations had already run. The migration runner needed a reliable sign that the historical schema existed.

My first sentinel used a table created by the last historical migration. That failed in a clean database test because the local bootstrap path also created the table from the TypeScript schema. The runner saw the table, inserted ledger rows for all historical files and skipped the SQL that created the restricted application role. It later failed when it tried to grant permissions to that missing role. The baseline rows had already been committed, so another run skipped the same files again. Repair required manual correction of the false ledger state.

I considered using the restricted role as the sentinel. PostgreSQL roles belong to the cluster, so a role created for another database on the same server could satisfy that check. The third candidate used a row-level security (RLS) policy that the final historical SQL migration created. PostgreSQL stores a policy against a table in one database. The project's TypeScript schema did not declare this policy, so this project's push bootstrap did not create it. That project-specific difference made the policy a useful sign that the historical SQL set had run.

The first implementation still contained a gap. It marked every SQL file currently present in the repository as applied when it found the policy. If a new migration arrived before a database ran the tracker for the first time, the runner would mark that new file as applied without executing it.

A later correction added the missing boundary. The final design stores the filename of the last historical migration that predates tracking. When the policy exists, the runner baselines files only up to that fixed cutoff. Every later file still executes through the normal transaction path.

const BASELINE_LAST_MIGRATION = "019_last_historical_migration.sql"

const baselineFiles = files.filter(
  (filename) => filename <= BASELINE_LAST_MIGRATION
)

This comparison relies on the repository's zero-padded numeric filename prefixes. The policy answers whether the historical schema exists. The cutoff defines exactly which files the runner may mark as historical. The baseline needs both facts.

What changed in the deployment design

drizzle-kit push remains useful for the development workflow this project chose, but the production migration job no longer runs it. Reviewed SQL files and the migration ledger own the production schema history.

The incident also changed how I treat interactive tools in automation. A CI command needs a documented non-interactive path for every decision it can raise. A timeout limits the wait, and its nonzero status must continue to block the deployment.

Schema-diff tools also need tests against the PostgreSQL features the application actually uses. For this incident, one relkind filter excluded the partitioned parents from the introspection result while returning their children as tables. A normal unpartitioned development database would not expose that failure.

Finally, a baseline needs two independent decisions. The sentinel identifies the existing schema state. The cutoff limits the historical files associated with that state. Treating either answer as the other creates a ledger that can lie.

References

• drizzle-kit push, including the documented introspection flow and --force behaviour
• drizzle-kit generate, including rename decisions during schema generation
• PostgreSQL pg_stat_activity, including the meaning of an idle backend
• PostgreSQL explicit locking, including the lifetime of session-level advisory locks
• PostgreSQL pg_class, including the r and p relation kinds and relispartition
• PostgreSQL declarative partitioning
• PostgreSQL policy catalogue, which records policies against tables
• Drizzle issues covering manual unique-constraint input 4531, programmatic pushSchema input 4651, automatic approval 4921, and reported destructive --force selection 3209
• Issue 6093, which reports partitioned parents missing from introspection in version 0.31.10 and discusses later release behaviour