Accelerating release velocity with deterministic automation
Salesforce teams running something the size of AI Cloud lose a lot of deployment time to manual release governance. The old process put release managers in the middle, coordinating over Slack across separate dashboards (SLO availability, PagerDuty configs, FIT results), and sign-off on production readiness took hours or days.
The AI Cloud Infrastructure team built Luminary, a centralized automation platform that acts as a release control plane. Release validation now runs as code instead of being interpreted by a person, and system health metrics decide directly whether a deployment is allowed.
Replacing human-gated latency with programmatic enforcement
The bottleneck was service owners waiting for manual verification. The Luminary team built an automated approval engine that runs the verification logic on the spot.
The engine queries telemetry (e.g., Argus for availability), cross-references FIT results across service boundaries, and confirms escalation policies before promotion is granted. That logic lives inside the workflow engine, so nothing queues behind a reviewer.
Previous State: Serialized, Human-Gated Process (Hours/Days)
New State: Self-Service Model governed by Luminary (Seconds)
Engineers get approval the moment the codified readiness thresholds are met, so release velocity scales with system health instead of reviewer capacity.
Automated production readiness via dependency validation
In a deep microservice graph, one service can report healthy while its upstream or downstream dependencies are failing. Cross-referencing that by hand across monitoring dashboards did not scale.
Luminary puts automated dependency validation into the promotion workflow itself. It walks the dependency graph and reads live availability signals and FIT results for every linked service.
Promotion halts unless all dependent services meet the predefined readiness thresholds. Production readiness becomes a property of the whole environment, which holds operational risk down as the graph gets more complex.
Event-driven promotion state transitions
Handoffs between environments (e.g., Stage to Production) used to stall on context switching and manual polling. An engineer waited for the stage deployment to finish, switched back to Slack or the approval dashboards, and triggered the next step by hand.
Luminary now hooks into execution event streams (e.g., FUN execution events). It watches deployment transitions as they happen and fires the next approval phase as soon as the previous stage completes successfully.
The same event-driven logic handles every transition (Stage -> Production, Production -> GIA), so the workflow keeps moving on telemetry instead of on somebody watching a dashboard.
Traceability through automated artifact generation
Tagging Git releases by hand left real gaps: which commits reached production, and which execution ID belonged to a given incident. The metadata was often incomplete or missing.
Luminary now generates the artifacts:
- Commit aggregation: it gathers every commit since the last release.
- Structured release notes: it generates the notes with that metadata attached.
- Execution reference: it attaches the deployment execution ID to the Git artifact (major, minor, or patch).
The artifacts come out consistent and traceable, which is what audits and incident correlation need. In regulated environments like GovCloud, Luminary also validates the required Change Cases and generates and populates GUS work items for the GIA channel, so those handoffs come out the same shape every time.
Centralized state management for service diversity
Over 100 services, each with its own dependencies, SLO definitions and testing criteria, need one configuration source. Luminary uses a centralized Postgres state engine as the source of truth for service metadata, dependency maps and policy definitions, pulling authoritative data from the monitoring systems.
That state model holds both static service attributes and live telemetry, so workflows adapt without an engineer editing per-service configuration during an approval.
Decoupling background processing for stability
Early versions ran the Flask API server, which handles the Slack interaction, alongside the background processing: telemetry synchronization and cron jobs. As automation load grew, the two contended for resources and the service got unstable.
The refactor introduced Starfall, a Golang engine dedicated to the heavy background work, including database synchronization and scheduled tasks. That pulls the data processing off the real-time Slack interaction layer. Postgres is still the state backbone, and Starfall keeps the data fresh asynchronously, which holds response times steady under concurrent load.
Leave a Comment