Skip to content
Aryan Deshmukh

    DevOps

    The Restart Command That Only Worked When a Human Typed It

    A supervisor noticed the crash, ran the restart and failed every time. The command was right. The shell it ran in was not.

    Read the story on Medium

    The same restart command, typed by a person and run by a supervisor. Only one of them ever worked.

    A background worker on one of our servers crashed because Redis refused a connection for a moment. A process supervisor was watching for exactly that. It noticed the worker was gone and ran the restart command it had been given.

    The restart failed. So did the next one, and every one after it, each ending in the same status line: Execution failed | Does not exist.

    The command itself was fine. Its two steps are the ones anyone on the team types to start the worker by hand: change into the app directory, then bundle exec sidekiq. Typed at a prompt, they pick the right Ruby and start cleanly. Run by the supervisor, the same two steps failed every time, and the log never said why.

    In this post

    1. 01Running the supervisor's command, exactly as written
    2. 02A shell does more at startup than it looks like
    3. 03Why every check said it was fine
    4. 04Why the supervisor couldn't say what was wrong
    5. 05The fix
    6. 06What transfers

    Continue reading on Medium