Rails
The Fix That Only Reached Half the Server
We fixed a config file, restarted the app and watched one feature recover. The other kept failing for a day and a half.
On the evening of the fix, two admin screens in our Rails app were uploading Excel sheets to the same S3 bucket. Same uploader class, same config file for its credentials. When we corrected that file, one screen started working on its very next request.
The other failed again that evening, and again the next evening, with the file correct on disk the whole time. No stale branch, no cached build, no second server. One file, one codebase, one machine, two different answers.
It turned out to be a very ordinary assumption breaking, one most Rails teams carry without ever testing it: "I restarted the app, so the app has the new config."
In this post
- 01A new feature's first evening in production
- 02The upload that looked like a casualty
- 03Same file, same symptom, a day later
- 04"Read once at boot" means once per process
- 05The deploy script already knew
- 06Why nothing told us
- 07Why it had never happened before
- 08The fix
- 09What transfers
- Ruby on Rails
- Sidekiq
- Debugging
- DevOps
- Software Engineering