Crazy Ivan

We recently had a departure of a senior resource that prompted us to go through all of our administrative passwords for the (frankly, surprising number of) systems we manage and update them.

The great fear and apprehension we and everyone feels about changing root or admin passwords is that it's never really crystal clear - I'm talking about vendor documentation in particular - what might break when you do. Like many small-to-mid size shops, we don't do this very often, because it doesn't add money to the bottom line and we have more work than we can handle just keeping the important stuff running smoothly.

But there is value in the exercise. Not for the stuffed-shirt security-Nazi / audit-police reasons, but because it's easy to lose sight of hundreds and thousands of incremental additions and changes to the network - even if you have a careful change control process.

So we did the password change, and by and large, we did a good job of identifying important systems we knew to be using the passwords and prepare them accordingly. In one instance, we missed one, but knew exactly what was wrong and were able to quickly find where it had stored admin credentials. Another surfaced later in the week, that shouldn't have been using admin credentials at all. Sometimes, if you're not real careful, a lab effort can go so well that you just move straight into production rather than re-build everything from scratch. Time is money, after all. Easy enough to fix, create a new set of credentials for that system and move on.

That latter scenario - something breaking that never in a million years would have been expected to be using admin credentials - repeated itself twice. The people responsible for it are, interestingly, the people responsible for us wanting to change the passwords in the first place.

In the movie "Hunt for Red October", the captains of Russian submarines would spontaneously make a sharp turn one direction or the other. This prevented them from getting a false sense of security - the natural tendency when everything is going well to consider it a result of intention rather than chance. "If it ain't broke, don't fix it." The maneuver was called a "Crazy Ivan", and an experienced U.S. submarine crew knew to expect it and keep following undetected - but only if they knew which way to turn.

If it's been a while since you've changed admin passwords, consider doing a Crazy Ivan of your own - not because you should distrust employees, but because you should distrust your ability to remember whether or not everything you put into production is following best practices.