Context and scale
Box edition of Bitrix24 with sales, procurement, warehouse and service on smart processes, ~30 portal apps and vendor modules; exchange with 1C:UNF through a closed vendor connector. Documented period: Aug–Oct 2026.
Problem
1.88M events were queued for dead 1C endpoints and growing by ~50k a day; three parallel Bitrix24 → 1C channels; an incident had created 329k empty records and ~358k unfinished workflows; 1,659 phantom serial-item cards; receipts linked to deals in 5 of 176 cases.
Analysis and options
Rebuilding the exchange on the 1C side was not possible: the connector is closed vendor code. Replaying 1.88M events would have pushed stale changes into accounting. Fixing on the portal side — subscriptions, filters, data — was reversible and fast.
Key decisions and trade-offs
- Drop dead subscriptions, do not replay. Events for endpoints nobody reads carry no value. A backup of the removed subscriptions keeps the step reversible.
- Send only finished documents to 1C. Own event filter: deals, requests, receipts and shipments go to accounting only when completed or posted. Fewer conflicts, slightly later data in 1C.
- Recycle bin, never hard delete. Dry run first, re-check of the condition before each record, JSON backup; 14 records kept because live deals referenced them.
What was done
- Subscriptions 159 → 54; queue emptied, new events read immediately; lost events in request handlers fixed.
- Phantom cards 1,659 → 681 (978 to the recycle bin), cards without a warehouse 1,079 → 101, orphaned reserves 16 → 0, stock reconciled with receipt documents and rollback.
- Incident aftermath cleared: workflows ~358k → 2.5k, empty records 329k → 529.
- Contract ↔ deal sync both ways, a 5-phase sales funnel with a manager report, receipt stage moves on posting, a portal health page.
- Service and warehouse workplaces with mobile PWAs, route planning, inventory, automatic receipts.
Result
The exchange with 1C works again and carries only finished documents; stock has been reconciled with receipt documents, with a rollback path for every step.
How it was verified
Queue size measured twice 45 s apart before the fix; after it, live polling by 1C and batch checks; every deletion preceded by a dry run and logged with a backup. The audit itself was reviewed by an independent LLM judge.
What I would do differently
Queue size and the number of unfinished workflows should be on a health page from day one — both problems grew for weeks before anyone looked.