Bureau Reporting Modernisation

[ System architecture: SFTP-based legacy pipeline → REST API data-sharing layer / before → after ]

~40%

Faster report processing after migrating from legacy to modern infrastructure

~99%

On-time file delivery after SFTP → REST API migration

99.9%+

Reporting platform uptime through failover mechanism implementation

Context

American Express's bureau reporting system, responsible for sending credit data to the major credit bureaux, was running on aging infrastructure built around SFTP-based file transfers. Reporting accuracy and timeliness are regulatory obligations; any failure has compliance and reputational consequences. The system had accumulated years of technical debt and operated with minimal failover capability.

Problem

The legacy pipeline had two compounding problems: a slow processing layer that created queue backlog on high-volume days, and an unreliable SFTP delivery mechanism that caused intermittent late or failed submissions. Regulatory reporting latency was becoming a risk. Meanwhile, the team's sprint velocity had plateaued and the say-do ratio was poor: backlog chaos was absorbing capacity that should have gone to the modernisation.

Approach

Separated delivery reliability from infrastructure modernisation

The instinct was to do everything at once. I pushed back: fix SFTP reliability first (quick win, lower risk), then modernise the processing layer. Sequencing matters: stakeholders see results earlier, and you don't bet compliance on a big-bang migration.

Scoped the REST API migration as a parallel-run before cutover

Designed a shadow-period where both SFTP and REST API pathways ran simultaneously. Compared outputs for parity. Only cut over when 30 consecutive business days showed zero discrepancy. Killed the SFTP dependency cleanly, not nervously.

Implemented a failover mechanism to close the uptime gap

Worked with engineering to spec a failover that automatically rerouted processing on primary node failure. Defined the SLAs, the alert thresholds, and the runbooks. This was the piece that took uptime from "mostly okay" to 99.9%+.

Rebuilt the team's delivery process from the backlog up

Ran a backlog audit: removed stale items, clarified acceptance criteria on everything in the next two sprints, and introduced a standing refinement session. Sprint velocity improved 25% YoY. Say-do ratio went from roughly 60% to 90%+ within two quarters.

What I'd Carry Forward

Sequencing is strategy. The order you tackle problems in often matters more than the solutions themselves, especially when compliance is involved and there's no margin for error.
A parallel-run period feels expensive until you compare it to a failed cutover. For regulated data, the shadow period is non-negotiable.
Velocity problems are almost always backlog problems in disguise. Fixing the upstream (clarity, sizing, sequencing) fixes the downstream (delivery, confidence, morale).