In 2023, our core platform was a 340,000-line PHP monolith. It served 12,000 active users, processed roughly 40,000 daily jobs, and had become the kind of codebase where every deployment felt like defusing a bomb. We knew we had to modernize. We also knew that "monolith to microservices" stories are usually told after everything works, without the messy middle.
This is the messy middle, told honestly.
01Why we outgrew the monolith
The signs were obvious for at least a year before we acted. Deployments took 45 minutes and required a full team on standby. A change in the billing module could break the notification service. Our database was a single 400 GB MySQL instance with 220 tables — no schema felt isolated from any other schema.
The breaking point came in March 2023. A junior developer made a one-line change to a query in the reporting module. It cascaded into a 4-hour outage during peak traffic because the query locked a table the payment flow depended on. We lost revenue, we lost trust with two enterprise clients, and — more importantly — we lost the assumption that our architecture could keep scaling with the business.
That week, we made the call: we were going to migrate, and we were going to do it properly.
02What "event-driven microservices" actually meant for us
We didn't want to just split the monolith into a dozen small services and call it "microservices." That's a distributed monolith — all the pain, none of the benefit. The real architectural shift was moving from synchronous request/response to event-driven communication.
The core idea: services publish events when something happens, and interested services react. When an order is placed, we don't call five services in sequence. We emit an order.created event, and the billing, notification, inventory, and analytics services each handle it independently.
That decoupling is what finally freed our teams to deploy independently.
03The four phases of our migration
We planned it as a 12-month migration. We finished in 9. Here's how it actually went.
Phase 1 — Strangler pattern, but slow
We used the Strangler Fig pattern: put an API gateway in front of the monolith, then gradually peel off functionality into new services. The first candidate was the notification system — it was low-risk, had clear boundaries, and if it broke, nothing catastrophic would happen.
We built it in Node.js, deployed it to a single ECS task, and routed all notification traffic to it. That worked. We celebrated.
We should not have celebrated. Within two weeks we discovered that notification logic had 47 hidden dependencies on other parts of the monolith, from user preference flags to template rendering. We had to keep the monolith running in parallel for those 47 dependencies while we decided what to do with each one.
This is where most migrations fail — not because microservices are hard, but because monolith boundaries are lies.
Phase 2 — The event bus becomes real
We introduced Amazon EventBridge as our event bus. Every service published structured events. But we made a critical mistake here: we didn't enforce a schema.
Within a month we had events with slightly different shapes flying around. Some had user_id, some had userId, some had nested objects where others had flat. We spent more time writing adapters than building features.
We stopped everything and spent two weeks standardizing on a schema registry. That decision alone saved us months of pain later.
Phase 3 — The distributed database problem
This is the part nobody warns you about properly. We knew we'd eventually need to split our 400 GB MySQL instance. What we didn't anticipate was how painful it would be to keep six services consistent while the data was still technically in one database.
Our solution was a mix of patterns:
- Saga pattern for multi-step transactions (order → payment → inventory → fulfillment)
- Outbox pattern to guarantee events were published when a database transaction succeeded
- Eventual consistency everywhere the user wouldn't notice it (analytics, notifications, reports)
- Synchronous calls only where truth mattered immediately (auth, payment capture)
- Structured JSON logging in every service, shipped to CloudWatch Logs Insights
- Distributed tracing with AWS X-Ray — every request got a trace ID that propagated across services
- Metrics on every event handler: throughput, error rate, and processing time
- Alerts for any event that failed three times in a row
- Deployment time: 45 minutes → 4 minutes
- Deployment frequency: 2 per week → 14 per week
- Mean time to recovery (MTTR): 3 hours → 22 minutes
- Infrastructure cost: +38% (new services, event bus, tracing, observability)
- Team size: unchanged (6 engineers)
- Production incidents: down 60%
- Pages per engineer per week (on-call): 4 → 1
- Don't migrate because it's trendy. Migrate because your monolith is actually blocking you.
- Strangle the monolith slowly. Big-bang rewrites fail. Make it a slow, low-risk progression.
- Invest in observability first. It's the difference between a migration and a disaster.
- Schema everything. Events, logs, configs. Enforce it.
- Plan for failure. Events will duplicate. Handlers will fail. Data will lag. Design around it.
We kept the MySQL database for six months into the migration. Splitting it too early would have been a disaster.
Phase 4 — Observability or death
Around month five, we had a distributed system that worked in staging and behaved unpredictably in production. Debugging was miserable. A single user request touched four services, and none of them logged consistently.
We invested heavily in observability:
This was the single highest-ROI investment of the entire migration. Without it, we'd have been blind.
04The two incidents that almost made us quit
We had two production incidents during the migration. Both were preventable in hindsight.
Incident 1 — The event storm
In month four, we shipped a change to the inventory service. A bug caused it to republish an event every time it received that same event type. Classic loop. Within 90 seconds, we had 2.1 million duplicate events on the bus. Downstream services collapsed. The payment service backed up. The notification service sent the same email 400 times to some users.
Fix: We implemented idempotency keys on every event handler and rate limits on the EventBridge rules. We also added a circuit breaker that would disable any event source producing more than 1,000 events per second.
Lesson: never trust that an event will only arrive once. Design for at-least-once delivery from day one.
Incident 2 — The silent data loss
In month seven, we discovered that 340 orders had been processed but never appeared in the analytics pipeline. The order service had published them correctly. The analytics service had subscribed. But a subtle bug in its event parsing had silently dropped events it couldn't parse — with no error thrown.
Fix: We made the parser strict (throw on unknown fields) and added a Dead Letter Queue (DLQ). Any event that fails processing three times goes to the DLQ. An engineer reviews the DLQ daily.
Lesson: silent failures are worse than loud ones. Make your errors visible, always.
05The numbers after nine months
By the end of the migration:
The cost went up. That's the honest part. But the reliability and team velocity made the trade worth it. We could now ship features that used to take a quarter in a single week.
06What we'd do differently
If we were starting again, three changes would have saved us at least three months:
1. Invest in observability before the first service, not after.
We spent weeks guessing at bugs we could have seen instantly with tracing and structured logs from day one.
2. Enforce the event schema from the very first event.
Standardize on a schema registry, version every event, and reject unknown fields. This prevents weeks of adapter code later.
3. Migrate the database last, not in parallel.
Trying to split the database while also splitting services multiplied our risk. If we'd kept MySQL intact for the first year and moved services around it, we'd have slept more.
07Is it worth it?
For our stage — 12,000 users, 40,000 daily jobs, growing — absolutely yes. But I want to be clear: microservices are not a magic upgrade. They're a trade. You trade deployment speed and independent scaling for operational complexity.
If we had been a team of 3, serving 500 users, this migration would have been a mistake. We'd have added 30% operational overhead for benefits we didn't need.
The right question isn't "should we use microservices?" It's "what problem are we solving, and does this architecture solve it?"
For us, the problem was clear: deployment velocity and blast radius. Microservices solved both. They also introduced new problems we hadn't anticipated, and we spent real money and real time solving them.
That's the honest story of our migration. If you're considering something similar, I hope some of this saves you a few of our mistakes.
08If you're thinking about migrating
If you're on a similar journey, here's what we'd tell you:
The migration we did wasn't elegant. It was messy, expensive, and full of surprises. But it was worth it. And if you're considering it for the right reasons, it probably will be for you too.
