A tool for tracking changes to production environments.

Need

Software changes frequently, especially hosted (SaaS) software.

Each change introduces the chance of breaking something. Rather than avoiding change, it benefits us to embrace change, but keep track of each change so that we can pinpoint the causes of breakages.

Changes include, but are not limited to, the following:

• Deploy
• Rollback
• Configuration change (via e.g. environment variables)
• Infrastructure change
• Changes to feature flags
• Rollout percentage change

Approach

Create a hosted web app that is essentially an append-only collection of structured log entries.

Each log entry has the following data:

• Message
• Timestamp
• Initiator: Who initiated this change?

Log entries can be added in two ways:

• via a user interface
• via an API (this would be useful for automatically recording deploys and rollbacks).

Future work could include

• adding integration with Slack (or similar communication tool) for easily adding new log entries.
• splitting up Initiator into actor (e.g. GitLab) and on-behalf-of (e.g. me).
• adding a way of attaching comments to log entries, to clarify after-the-fact what happened and why.

Benefits

There is a single place to see all changes that have been made to production, which makes investigating breakages simpler.

Drawbacks

• Implementation effort
• Adoption effort
• Ongoing maintenance (and ownership) to the hosted web app