Some sort of memory leak with tranquil and postgres
did:plc:jrrhosrfzgjf6v4oydav6ftb opened this Apr 8, 2026 8 comments
did:plc:jrrhosrfzgjf6v4oydav6ftb opened Apr 8, 2026
Just experienced my PDS and the VPS running it becoming unusable due to what seems to be a memory or process leak of some kind. I've had the VPS experience similar symptoms a few times in the past few weeks, so this doesn't seem to be a one-off problem (I didn't look into it back then because I was busy at the times they occurred, but likely the same cause).
Investigating shows that the tranquil-pds process was using almost 50% of RAM on the machine (2GB total), and a whole ton of postgres-wrapped (name is due to NixOS stuff) processes on the tranquil-pds Postgres db & user.
Additionally, the logs for tranquil-pds before shutting it down show a TON of slow queries. Though I'm not sure if they're just a symptom of the memory leak or the cause.
Apr 08 11:46:05 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:05.916196Z INFO tranquil_pds::api::proxy: Proxying GET request to https://api.bsky.chat/xrpc/chat.bsky.convo.getLog?cursor=2222224he6tcn
Apr 08 11:46:20 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:20.978686Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=3.156485986s elapsed_secs=3.156485986 slow_threshold=1s
Apr 08 11:46:21 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:21.586919Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=2.835107378s elapsed_secs=2.835107378 slow_threshold=1s
Apr 08 11:46:21 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:21.916735Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=3.927041836s elapsed_secs=3.927041836 slow_threshold=1s
Apr 08 11:46:22 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:22.430020Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=4.195293246s elapsed_secs=4.195293246 slow_threshold=1s
Apr 08 11:46:23 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:23.161315Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=3.376707243s elapsed_secs=3.3767072430000002 slow_threshold=1s
Apr 08 11:46:23 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:23.639974Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT cid, data FROM …" db.statement="\n\nSELECT cid, data FROM blocks WHERE cid = ANY($1)\n" rows_affected=117930 rows_returned=117930 elapsed=5.195079337s elapsed_secs=5.195079337 slow_threshold=1s
Apr 08 11:46:29 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:29.662683Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.582309402s elapsed_secs=1.582309402 slow_threshold=1s
Apr 08 11:46:36 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:36.510395Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=2.534972409s elapsed_secs=2.534972409 slow_threshold=1s
Apr 08 11:46:44 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:43.994689Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=7.960787661 slow_acquire_threshold_secs=2.0
Apr 08 11:46:49 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:49.225832Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=7.485912726s elapsed_secs=7.4859127260000005 slow_threshold=1s
Apr 08 11:46:55 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:54.314333Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=5.720312542 slow_acquire_threshold_secs=2.0
Apr 08 11:47:00 pointbreak tranquil-server[1480449]: 2026-04-08T01:46:59.704685Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=7.076086058s elapsed_secs=7.076086058 slow_threshold=1s
Apr 08 11:47:05 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:05.097688Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=6.354272873s elapsed_secs=6.354272873 slow_threshold=1s
Apr 08 11:47:06 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:06.433738Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.524335997 slow_acquire_threshold_secs=2.0
Apr 08 11:47:13 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:13.351688Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.791472538s elapsed_secs=1.7914725379999998 slow_threshold=1s
Apr 08 11:47:19 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:18.785937Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=3.993121793s elapsed_secs=3.9931217930000003 slow_threshold=1s
Apr 08 11:47:21 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:20.982497Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=2.953758261s elapsed_secs=2.953758261 slow_threshold=1s
Apr 08 11:47:23 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:23.003639Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=2.22597497s elapsed_secs=2.22597497 slow_threshold=1s
Apr 08 11:47:25 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:25.391596Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.348651819s elapsed_secs=1.348651819 slow_threshold=1s
Apr 08 11:47:27 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:27.025755Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=1.908714161s elapsed_secs=1.908714161 slow_threshold=1s
Apr 08 11:47:28 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:28.031694Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.131459332 slow_acquire_threshold_secs=2.0
Apr 08 11:47:29 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:29.535249Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.085629726 slow_acquire_threshold_secs=2.0
Apr 08 11:47:32 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:32.159421Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=1.143694857s elapsed_secs=1.143694857 slow_threshold=1s
Apr 08 11:47:33 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:33.503745Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.842146804s elapsed_secs=1.842146804 slow_threshold=1s
Apr 08 11:47:37 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:37.925769Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT oid::regtype::text, typtype, typcategory, …" db.statement="\n\nSELECT oid::regtype::text, typtype, typcategory, typrelid, typelem, typbasetype FROM pg_catalog.pg_type WHERE oid = $1\n" rows_affected=1 rows_returned=1 elapsed=1.562224917s elapsed_secs=1.562224917 slow_threshold=1s
Apr 08 11:47:38 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:38.704687Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=1.012291151s elapsed_secs=1.012291151 slow_threshold=1s
Apr 08 11:47:43 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:42.901572Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.004627818 slow_acquire_threshold_secs=2.0
Apr 08 11:47:50 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:50.884227Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT enumlabel FROM pg_catalog.pg_enum …" db.statement="\n\n\nSELECT enumlabel\nFROM pg_catalog.pg_enum\nWHERE enumtypid = $1\nORDER BY enumsortorder\n \n" rows_affected=4 rows_returned=4 elapsed=12.629846721s elapsed_secs=12.629846721 slow_threshold=1s
Apr 08 11:47:53 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:53.448991Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT enumlabel FROM pg_catalog.pg_enum …" db.statement="\n\n\nSELECT enumlabel\nFROM pg_catalog.pg_enum\nWHERE enumtypid = $1\nORDER BY enumsortorder\n \n" rows_affected=13 rows_returned=13 elapsed=1.148631597s elapsed_secs=1.148631597 slow_threshold=1s
Apr 08 11:47:56 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:56.143303Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT enumlabel FROM pg_catalog.pg_enum …" db.statement="\n\n\nSELECT enumlabel\nFROM pg_catalog.pg_enum\nWHERE enumtypid = $1\nORDER BY enumsortorder\n \n" rows_affected=4 rows_returned=4 elapsed=1.601305642s elapsed_secs=1.601305642 slow_threshold=1s
Apr 08 11:47:57 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:57.242686Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=22.13556927s elapsed_secs=22.13556927 slow_threshold=1s
Apr 08 11:47:59 pointbreak tranquil-server[1480449]: 2026-04-08T01:47:59.230598Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=4.299198743 slow_acquire_threshold_secs=2.0
Apr 08 11:48:00 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:00.332578Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.891885621s elapsed_secs=1.8918856210000001 slow_threshold=1s
Apr 08 11:48:05 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:04.728683Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=4.867173736 slow_acquire_threshold_secs=2.0
Apr 08 11:48:07 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:06.381218Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.9686508160000002 slow_acquire_threshold_secs=2.0
Apr 08 11:48:10 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:10.386955Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=4.680521942 slow_acquire_threshold_secs=2.0
Apr 08 11:48:14 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:13.042691Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=3.785203768s elapsed_secs=3.785203768 slow_threshold=1s
Apr 08 11:48:17 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:16.560147Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=4.10853552 slow_acquire_threshold_secs=2.0
Apr 08 11:48:19 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:18.822207Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=3.609882527 slow_acquire_threshold_secs=2.0
Apr 08 11:48:22 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:21.205220Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=2.178683201s elapsed_secs=2.178683201 slow_threshold=1s
Apr 08 11:48:25 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:24.370691Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=3.39899803s elapsed_secs=3.39899803 slow_threshold=1s
Apr 08 11:48:27 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:26.057686Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=3.002003344 slow_acquire_threshold_secs=2.0
Apr 08 11:48:28 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:28.031591Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=2.623957096 slow_acquire_threshold_secs=2.0
Apr 08 11:48:33 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:31.575764Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=4.157032744s elapsed_secs=4.157032744 slow_threshold=1s
Apr 08 11:48:38 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:37.882102Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=7.120300271 slow_acquire_threshold_secs=2.0
Apr 08 11:48:45 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:44.416011Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=7.198914919s elapsed_secs=7.198914919 slow_threshold=1s
Apr 08 11:48:50 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:47.325725Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.175709311s elapsed_secs=1.175709311 slow_threshold=1s
Apr 08 11:48:54 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:52.024688Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=9.881210468s elapsed_secs=9.881210468 slow_threshold=1s
Apr 08 11:49:01 pointbreak tranquil-server[1480449]: 2026-04-08T01:48:59.232680Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=8.208512939s elapsed_secs=8.208512939 slow_threshold=1s
Apr 08 11:49:12 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:02.033513Z INFO tranquil_sync::subscribe_repos: Client closed connection abruptly
Apr 08 11:49:12 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:02.580811Z INFO tranquil_sync::subscribe_repos: Firehose subscriber disconnected subscribers=18
Apr 08 11:49:22 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:14.076727Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=15.45022882s elapsed_secs=15.45022882 slow_threshold=1s
Apr 08 11:49:30 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:28.110682Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=26.439509376 slow_acquire_threshold_secs=2.0
Apr 08 11:49:37 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:31.917645Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=8.063040691s elapsed_secs=8.063040691 slow_threshold=1s
Apr 08 11:49:48 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:41.643443Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=11.513410579s elapsed_secs=11.513410579 slow_threshold=1s
Apr 08 11:49:55 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:50.675400Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=12.723399014s elapsed_secs=12.723399014 slow_threshold=1s
Apr 08 11:50:04 pointbreak tranquil-server[1480449]: 2026-04-08T01:49:59.361681Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=10.303268218s elapsed_secs=10.303268218 slow_threshold=1s
Apr 08 11:50:14 pointbreak tranquil-server[1480449]: 2026-04-08T01:50:06.645660Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=10.569036152s elapsed_secs=10.569036152 slow_threshold=1s
Apr 08 11:50:34 pointbreak tranquil-server[1480449]: 2026-04-08T01:50:15.659573Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=2.229890814s elapsed_secs=2.229890814 slow_threshold=1s
Apr 08 11:50:40 pointbreak tranquil-server[1480449]: 2026-04-08T01:50:35.906685Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=30.951963439s elapsed_secs=30.951963439 slow_threshold=1s
Apr 08 11:50:49 pointbreak tranquil-server[1480449]: 2026-04-08T01:50:42.869120Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.268296059s elapsed_secs=1.2682960589999999 slow_threshold=1s
Apr 08 11:50:59 pointbreak tranquil-server[1480449]: 2026-04-08T01:50:53.598632Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=20.40648406s elapsed_secs=20.40648406 slow_threshold=1s
Apr 08 11:51:02 pointbreak tranquil-server[1480449]: 2026-04-08T01:51:01.819683Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=11.325920789s elapsed_secs=11.325920789 slow_threshold=1s
Apr 08 11:52:09 pointbreak tranquil-server[1480449]: 2026-04-08T01:51:18.008784Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=16.521588592s elapsed_secs=16.521588592 slow_threshold=1s
Apr 08 11:52:17 pointbreak tranquil-server[1480449]: 2026-04-08T01:52:09.873738Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=65.659963708s elapsed_secs=65.659963708 slow_threshold=1s
Apr 08 11:52:24 pointbreak tranquil-server[1480449]: 2026-04-08T01:52:21.515067Z WARN sqlx::pool::acquire: acquired connection, but time to acquire exceeded slow threshold aquired_after_secs=14.709366053 slow_acquire_threshold_secs=2.0
Apr 08 11:52:36 pointbreak tranquil-server[1480449]: 2026-04-08T01:52:25.525398Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=7.519512403s elapsed_secs=7.519512403 slow_threshold=1s
Apr 08 11:52:50 pointbreak tranquil-server[1480449]: 2026-04-08T01:52:41.520141Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=17.405144986s elapsed_secs=17.405144986 slow_threshold=1s
Apr 08 11:53:00 pointbreak tranquil-server[1480449]: 2026-04-08T01:52:51.603772Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=17.211790815s elapsed_secs=17.211790815 slow_threshold=1s
Apr 08 11:53:09 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:03.173687Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=13.592001469s elapsed_secs=13.592001469 slow_threshold=1s
Apr 08 11:53:12 pointbreak systemd[1]: Stopping Tranquil PDS - AT Protocol Personal Data Server...
Apr 08 11:53:18 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:10.962715Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=10.632140503s elapsed_secs=10.632140503 slow_threshold=1s
Apr 08 11:53:32 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:24.774694Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=15.500868687s elapsed_secs=15.500868687 slow_threshold=1s
Apr 08 11:53:42 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:35.453689Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=1.209670591s elapsed_secs=1.209670591 slow_threshold=1s
Apr 08 11:53:53 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:45.092832Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=24.530986738s elapsed_secs=24.530986738 slow_threshold=1s
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:53:55.864239Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE comms_queue SET status …" db.statement="\n\nUPDATE comms_queue\n SET status = 'processing', updated_at = NOW()\n WHERE id IN (\n SELECT id FROM comms_queue\n WHERE status = 'pending'\n AND scheduled_for <= $1\n AND attempts < max_attempts\n ORDER BY scheduled_for ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING\n id, user_id,\n channel as \"channel: CommsChannel\",\n comms_type as \"comms_type: CommsType\",\n status as \"status: CommsStatus\",\n recipient, subject, body, metadata,\n attempts, max_attempts, last_error,\n created_at, updated_at, scheduled_for, processed_at\n" rows_affected=0 rows_returned=0 elapsed=1.660122712s elapsed_secs=1.660122712 slow_threshold=1s
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:02.953419Z INFO tranquil_server: Shutdown signal received, stopping services...
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:02.953512Z INFO tranquil_pds::crawlers: Crawlers service shutting down
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:02.954394Z INFO tranquil_ripple::gossip: gossip engine shutting down, flushing final delta
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:03.385688Z INFO tranquil_pds::scheduled: Scheduled tasks service shutting down
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:03.504228Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT data FROM blocks …" db.statement="\n\nSELECT data FROM blocks WHERE cid = $1\n" rows_affected=1 rows_returned=1 elapsed=21.902694327s elapsed_secs=21.902694327 slow_threshold=1s
Apr 08 11:54:03 pointbreak tranquil-server[1480449]: 2026-04-08T01:54:03.690778Z INFO tranquil_pds::comms::service: Comms service shutting downNo activity yet.