Wednesday, October 29, 2008

Novell Sees The Light on Maintenance

We processed our renewal today, and learned that Novell is now including unlimited support with the software maintenance MLA customers purchase each year. This prevents you from needing to buy Premium 1000 or 2000 (or more), and be worrying about how many times you call.

This is much more consistent with enterprise vendors like Oracle, IBM, and Sun, and is a welcome change. Unfortunately, it's a case of too little, too late. We're done waiting for Novell to do what customers like us have told them for 4, 5 or 6 years. As the saying goes down here, "Even a good dog will only stay on the porch for so long."

I'm told that more changes are coming over the next year, and I'm definitely interested in hearing what they are. I suspect it will address many of the complaints and confusing practices we've called out here, leaving only the question of "What took you so long?" for them to answer.

Tuesday, October 14, 2008

OES Linux and 8 Character Passwords

Did you know that OES on SLES is incapable of using passwords for the Root user longer than 8 characters?


Did you also know that OES on SLES allows you to blank out the Root user password using the passwd utility?


Did you know that lots of stuff is dependent on the Root user, and that password cannot be controlled through NAMCD or eDirectory?


Hoo boy.

Friday, September 12, 2008

Here It Comes (Microsoft Update)

Well, so much for saving money using Microsoft versus Novell. :-)

If ALL we wanted to do was apples-to-apples functionality, and had no needs for the more advanced (and to us, compelling) SharePoint features, we'd probably be able to do more for about the same spend.

But we DO need those Enterprise features, and that means the E-CAL suite. And that means we're definitely spending more than we used to.

What a lot of antagonists of Microsoft don't understand, for all their yelling and fear-mongering, is that the money you need to run Microsoft products is still barely a blip for big companies.

Novell then, as victims of their own success, have relegated themselves to the small-business/school/government niches where they've always done well - places where price trumps value.

Where companies make decisions based on value - not just price - Microsoft wins, hands down. Even if it's more expensive than Novell, it's still cheap. I can run our enterprise on about $70,000 worth of Novell licensing a year, including their pathetic excuse for phone support. Our Microsoft bill is likely to be around 4 times that, but that includes Office Pro Plus, and the entire E-CAL suite. Compare that to what a company spends per-head on enterprise systems like Oracle, etc., and it's a drop in the bucket. More people will get more productivity and efficiency out of MS products at that price than they ever will from their ERP system.

Anyway, we're looking forward to getting up to speed on the MS technologies - we're having in-house training performed, tailored to our specs and individuals, by an MS certified education partner. Other organizations here in town have done the same with this company, and have been very pleased.

If only Dell would release their E-Series laptop, we could really get moving with this effort.

Friday, August 22, 2008

VMWare and NFS on NetApp Filers

Ran across a very disturbing issue this week, which caused the corruption of nearly half of our virtual machines - certainly all of the ones which were experiencing any load to speak of.

If you run VMWare ESX 3i against a NetApp filer using storage pools defined as NFS mounts (of which a given cluster can only have 32, by the way), and you're a good Systems Engineer and follow the NetApp best practices guidelines for configuring ESX to work optimally with their products, then you'll turn on this nifty little switch called NFS.Lock.Disable by changing the default value of "0" to "1".

The document in question from October, 2007, used to be on the NetApp site (and maybe VMware's as well), but is now relegated to one of those cool sites that leech stuff from the web and hold it forever.

Then about 10 months later, or 2 months ago, it changed.

Suddenly, no mention of NFS.Lock.Disable.

This document, on the VMware website, however interestingly titled, makes no mention of it the setting. It's authored by NetApp as well, and is dated May 2007.

So what gives? Why so many documents and versions on the same topic, by so many authors? Who knows. All we know is that "best practices" for VMWare ESX 3 customers using NetApp and NFS storage pools was to set NFS.Lock.Disable to 1.

The problem is that VMWare ESX 3i isn't what ESX 3 used to be. From the poor retention of logs (not my opinion; statement of fact from VMware support), to the singular dependency on file locks to ensure split brain conditions don't occur, all the way to patches that somehow slip past QC with license-detonation code included.

All of this means when you do what VMware and NetApp told you to do in order to deploy their products together successfully, you put your data at risk.

And we did, and ours was, and it sucked.

Split brain means, in essence, that the system in charge of deciding what VM goes where (VCenter in this case) is unaware or uncertain of what ESX host holds which guest operating systems. In this condition, two separate ESX hosts can be - simultaneously - running the exact same guest OS instances. The bad news is, as you might imagine, that one half of that "brain" doesn't know what the other half is doing. What results is the utter annihilation of your filesystem, and depending on where and how you keep files, a very long process of restoring to a known good state.

The symptoms are beyond bizarre. VCenter shows a guest VM as being on a different host just about every second. Opening up a VM's console may give two people completely different screens, because you're actually looking at different "real" instances of the same virtual machine. Shutting down VCenter doesn't make things better; connecting directly to the ESX host will show you a wildly fluctuating number of guests running.

The only remedy is to use some neat "unsupported" (wink wink) console commands on the ESX hosts, and 'kill' the offending VM's. The faster you do that, the less badly your data will be matter how you slice it, it stinks.

VMware's Platinum support was surprisingly disappointing. Our first tech "hadn't been trained on 3i yet", and it took a while to get to someone who was. Like, hours. The RCLI is lacking commands that ESX 3 used to have, which vexes them. Following a host reboot, logs aren't kept. At all. They're not kept at all. What? Yes. Used to be in 3, but not in 3i. Nice. Once we got back up, it took ages and a lot of pretty irate e-mail to get someone to do some post-mortem analysis. Ultimately, we heard the details here straight from the horse's mouth. They're trying to eradicate the versions of that document that advise NFS.Lock.Disable - word never made it to us, somehow. They say they've known about the problem for about 30 days, which seems unrealistic. They say about 12 customers have had the same exact issues. And, unequivocally, they say to turn off that NFS.Lock.Disable shit, post haste.

My hope is that people who have deployed with this setting seriously consider changing it, and perhaps ask NetApp WTF? (or better yet, share their experiences in the comment area below). And additionally, my hope is that we're able to convince VMware that the deficiencies with 3i are totally unacceptable (no matter how insignificant they may seem to those with direct-attached or FC-attached disk). They could take a cue from Novell regarding heartbeat and keepalives to make sure direct host-to-host communication is used as a failsafe against these goofy file locks before allowing a VM to start on a new host.

Tuesday, July 01, 2008

Where Apple Won't Go

I remain befuddled by Apple Computer's outright thickheadedness with regard to their products and the American enterprise.

It's plain for all to see that Apple want absolutely no part of selling into corporations other than boutique art houses, or any other non-consumer segment other than schools. Yet despite this, Apple publishes a link to a story - on their own Apple "Start" page no less - highlighting the fact that 8 in 10 companies in America have Mac computers in production.


So, which is it - you don't care about corporations, or you're proud of the penetration your products have made into the enterprise space?

If it's the latter, how about actually ramping up an enterprise business unit? You know, with financing and on-site support programs and all the rest, like the other big kids?

Apple has done so poorly at speaking to the enterprise, that a group of five companies this week decided they'd do it for them. I love one of the quotes in the story.

"There's an information vacuum that we want to fill," T. Reid Lewis, president of Group Logic, told InformationWeek.

That's for damn sure.

Given that the newest iPhone is touted as having "enterprise" hooks, vis-a-vis integration of sorts with Exchange mail systems (not exactly in-use at many homes), one can hope that Apple's efforts to get into the enterprise will become a bit more purposeful than the X Server or X San. They need only the slightest breath to break the logjam holding Mac computers back from overrunning the enterprise space. That they pay us no attention is becoming less of an interesting quirk, and more of an insult.

Thursday, June 26, 2008

Here It Comes

With a requirement in-hand for a business system that requires Microsoft products on the back-end, we've begun our analysis of a Microsoft-centric technology stack versus the Novell solution we have in place today.

The results thus far have been pretty surprising. I could actually save money by going with Microsoft products to meet our requirements, and increase the functionality we provide our users. The annual maintenance-only costs for MS licenses (the Core CAL) is about half of what Novell's NOWS maintenance runs. The cost of the license is a bit higher, but is masked by the ability to finance the license cost for free over a 3 year term. The net result is that, year over year, I'd spend less with Microsoft for equivalent functionality, and would get SharePoint for free - which people are almost yelling for around here.

Microsoft has done a lot to improve the quality and stability of their products, as well as to bring their features closer to level with those to which we're accustomed in the Novell realm. When you look at all of the third-party integration points, and the benefits to be extracted by leveraging them, it's a very compelling proposition.

Those who know me will know that it's a very painful thing to admit, but it's no secret that I consider the Novell of my past to be long since dead. The people who knew how to envision, build, and maintain these products are long gone. Before much longer, we won't be able to get any help migrating, and doing so is an inevitability for us. I wish those who can continue on as they are the best of luck, but fighting it just for the sake of fighting it is a proposition that adds no value.

Wednesday, June 04, 2008


Look, I'm really proud that your organization has deployed Office 2007. Now get over yourselves and change the default file format back to Office XP / 2003 compatibility mode, so that you don't make your customers and external colleagues feel like idiots for asking "What's a docx (or xlsx) - can't you just send it to me as a Word doc (or Excel sheet)?"

ESPECIALLY for Mac users, who don't have the luxury of an Excel or PowerPoint "compatibility pack" in Office 2004.

Thank you,
The World

Sunday, June 01, 2008

Tough loss

This week saw me experience the first ever of my employees to resign. Tough, tough loss.

I've had Help Desk contractors who have left to do other things, which is what I prepare them to do since there are rarely opportunities for them to come on-board.

I've had employees that I've wanted to fire, but thankfully haven't, because it helped me get better at coaching problematic individuals and poor performers.

This one was a high performing, ambitious, young professional. If one should always be looking for their replacement as a way to facilitate their own upward mobility, this person was to be my replacement.

I shouldn't be surprised, given that their departure is identical to a career mistake I wish I could take back. Hubris. I was going to get my way come hell or high water, and if it meant I had to quit to show how serious I was, I was going to do it. So, out the door I went to a 45% raise and better vacation and benefits and less stress. And I liked it for exactly 9 months, at which point I began to loathe it.

This employee has taught me a lot over the 5 years I've mentored him. They challenged me when I've been less than intellectually honest, when I've been less than engaged, when I've bordered on doing something rash in the name of righteousness, when I've been unnecessarily rigid in my beliefs or decisions.

Leading this individual has also reinforced to me some lessons I had already learned.

Problems will always get fixed. Maybe not on the time table you'd prefer, and maybe not without having it brought to light and guarded repeatedly, but they do get fixed.

The grass is never, ever greener. The only reason this person went to another employer was because it was different. They might as well have gone to McDonald's. I've heard the pitch, the rationale, etc. - it doesn't add up. Bottom line, whether or not they were right, they believed that they were undervalued and that this fact would never ever change. Sounds awful familiar.

Hopefully I'll be able to remember and believe these lessons over the next 5 - 7 years, because there are periods of time where it's very, very difficult to keep going down this same road. I know that doesn't make my condition unique. I'd be perfectly justified in leaving, just as my departed employee was. That doesn't make leaving a good idea.

Now I have to identify who will be the next leader - either someone I have working for me already, or someone who we bring on-board specifically for the purpose. It would be tough for me to go somewhere else at this level never having lost a key employee. I'm actually looking forward to the opportunity to use this event to change some things for the better.

Wednesday, May 14, 2008

Hoo boy...

We're mid-way through our VMWare implementation project. It's been a very good learning experience for everyone. A couple of minor hiccups with an application here or there, but nothing unmanageable.

Part of this effort will involve migrating services currently running on NetWare 6.5, on to Novell's OES 1 SP 2 platform running on SLES...we've already replaced every other NetWare server in our environment with an OES server.

As you may recall from previous posts, we've been very disappointed with Novell's handling of patching for this "flagship", "enterprise class", "best-of-breed" operating system. In a word, it's a joke. Since we all value our jobs, we've decided it would be in our best interest NOT to adopt an automated or scheduled patching process for any of our OES servers...way too little quality control, way too much instability, and an utter disregard for the concepts of "stable" code bases.

Last night, that decision proved to be remarkably prescient.

We needed to see what happens when we migrate an NSS volume connected via iSCSI from a NetWare host to an OES/SLES host. So, off to the lab to build out a couple of simple servers. The NetWare box was built on a simple workstation PC in about an hour or so. The OES server took over 6 hours, and over 4 of that was waiting for the RUG update to finish (our standard process is to patch the server once, prior to productionalization, and never do it again once in the field).

We continued following our standard process for build-out after the first RUG update, which included kernels and all other manners of calamity. Once the second RUG update process completed, another reboot was in order.

Then it happened. TTY hadn't started. Absolutely nothing loaded that required a file system.

Upon further investigation, the SATA hard disks in our IBM x206 server - which, at install time, were designated as SDA1 and SDA2 by SLES, were now appearing as HDA1 and HDA2.

That's right. Running a patch process changed the paradigm for the hard disks in a running server, leaving the system unbootable.

Now, you might imagine how difficult it would be to address this issue - remotely - on 50 servers in the field. Even with the RSA cards we have and the boot-to-ISO capability, we'd be looking at untold hours wasted, recovering from an issue caused by Novell's complete and utter failure to act as responsible custodians for their product.

I'm now more convinced than ever that Novell lacks the leadership, and indeed the brainpower it once had, required to engineer and support an enterprise network operating system. The incredible legacy Drew Major created at Novell has been squandered, irrevocably. This is a company that has announced they won't have the first support pack to fix long acknowledge, significant issues in their latest version of the flagship operating system product until nearly 18 months after it's release.

What more reason do customers need to abandon Novell, with haste? All of the arguments of superiority Novell enjoyed at Microsoft's expense have been eradicated - not by Microsoft's improved product quality, but by the decline of Novell's product quality.

Certainly this will seem overly pessimistic, possibly bombastic, and over-reacting, to some of the more ardent red-blooded IT folks (and the indifferent). I am guilty of being passionate about the quality of service I deliver to my customers, and about holding others to the same expectations I have of myself - especially the vendors we've selected, especially when their performance goes into decline. This is the case far too often in our industry. It's inexcusable, and I cannot help but think a combination of Wall Street pressures, greed, and ignorance have all conspired to chip away at the foundation of our industry.

The cost and complexity of operating an enterprise IT infrastructure are growing exponentially, and in too many cases, completely without reason or justification. Nobody is concerned with making efficient, quality software any longer. Promise the world, ship tons of install DVD's, and if it doesn't work, shrug your shoulders and passively blame the customer for having either deficient requirements or deficient skill sets.

If our industry continues at this rate, the entire nation will have a very difficult time of competing abroad.

Thursday, May 08, 2008

Calling On Behalf Of...

I seem to be getting more and more telemarketing calls that start off with "Hi, I'm (insert name here) calling on behalf of (prominent vendor)."

Immediate bristling and rise in blood pressure occurs here.

I can't help it. It's just preposterous, to me. What can the hit rate for cold call prospects be - 0.1% tops? I have to suffer because a sales organization has run out of real work to do and has resorted to beating bushes and shooting at anything that flies out?

The puzzling part is that I have a Microsoft account team; I know who all of them are. They know what we need, and what is off-limits. I guess it's kind of an "oopsie" moment when I ask them to clarify what "on behalf of" means.

Me: "So you're with Microsoft?"
Them: "Yes."
Me: "Because I have a Microsoft account rep, who I just spoke to in e-mail yesterday about this. You're a Microsoft employee?"
Them: "On behalf of Microsoft, yes."
Me: "Well, I deal directly with a Microsoft account team. Thanks." (click)

The call today found me a bit less tolerant.

Them: "I'm so and so calling on behalf of Microsoft, working with the Exchange team, and would like to ask you a few questions."
Me: "Well, I deal directly with an account team at Microsoft; I don't deal with partners."
Them: "Well did you about Microsoft's security product called Frontline?"
Me: "Yes, we're not using it."
Them: "Well did you know that Microsoft has partnered with Nortel and (someone else)?"
Me: "They partner with just about everybody, yes."
Them: (pause) "Okay, then thank you."

Flabbergasting. What in the world can they possibly expect to come from a cold call, much less a cold call that has you hiding your own company's identity behind your partner's?

It's a good thing for Microsoft that there exist a lot of unknowledgeable, uncritical decision makers with large budgets and a susceptibility to FUD.

Wednesday, April 30, 2008

A Tale of Two Analysts

Now some 14 months post-release, it's abundantly clear to all but the most severely dunderheaded IT wonks that Vista will be on the podium of the Microsoft Failure Grand Prix. Not sure exactly where it will stand relative to Bob and Windows ME, but it's certainly not distinguished company to keep.

As I had stated (not predicted, because it wasn't a guess), in January 2007:

The first (well, pair) indicated that IT Execs weren't sold on Vista, and questioned whether or not Vista presented a realistic ROI case for companies.

I'll save you some reading. IT Execs aren't sold on Vista because it doesn't present a realistic ROI case.

Well, it turns out I was right. To wit, rumblings from the very top of Microsoft have been felt in faraway lands as rumors of Vista's successor are being released, and hints that VIsta should have been aborted mid-term slowly escape.

Dell, HP, and Lenovo have all been pressured by their enterprise customers into providing Windows XP via downgrade righs beyond the June 30, 2008 deadline Microsoft imposed for halting sales of XP. That's right, customers would rather use XP - even if they have to buy a premium Vista license to do it.

I don't know about Steve Ballmer, but that's how I spell failure.

More interesting was the completely divergent track two of the "industry's leading" analyst firms took in penning recommendations to enterprise customers regarding Vista.

The Gartner group proclaimed that Windows is "crumbling". That's right, Gartner actually said something disparaging about a very prominent technology product.

Forrester, it appears, prefers the Kasey Kasim approach - keep your feet on the ground, and yada yada yada.

"Vista is an inevitability, for a number of reasons," said Ben Gray, an analyst at Forrester Research Inc. He then ticked off several, including Windows XP's announced retirement and unsubstantiated talk about Vista's successor, Windows 7.

Mr. Gray invoked the wrath of Kahn in the comments section of this article. From being labeled as too ignorant to possibly hold the position he serves, all the way to questioning the integrity of Forrester. And rightly so. He is completely out of touch, and it is an embarrassment for Forrester to have this person writing on their letterhead.

One thing is perfectly clear - I was not alone in my view of how Vista would fare in the marketplace. It's done horribly, and Microsoft's attempts to pressure companies into using it have backfired. If they don't hit a home run with the next version of Windows, we may look back on Vista as the beginning of the end for Microsoft.

Tuesday, April 29, 2008

Two For Two

Two proposals for projects and capital expenditures in my new role, and two approvals. I've had very valuable help from my manager in fine-tuning (or totally revamping) parts of the proposals so that they're appropriate for the audience. We've done very well it would appear...despite a very tight economy and conservative holders of purse strings, we've approved nearly $250,000 of new projects.

The exercise of doing both is something everyone agrees should be commonplace for CapEx requests. I've been able to show both proposals will save real money, providing enhanced capabilities for free plus generate a return. It took a little effort and brainstorming, but it was far from impossible.

A lot of techies know when they've found the right thing to solve a given problem...the challenge is to get them to put it into terms that make sense to businesspeople. Technology hasn't been about 1's and 0's for a long time. If you want to be a technology professional, you need to link technology solutions to business problems, and sell them in non-technical terms using real numbers. If you can't do that, well, there's probably a call center hiring someplace.

Monday, April 21, 2008

Internationally Known

What a great tool LinkedIn has become for me.

Like anything, you get out of it what you put into it. I've taken some time to dredge up the names of faces I've met throughout the 15 years I've been a technology professional, and have been elated to find so many of them on LinkedIn. Reconnecting and catching up with former colleagues has been quite a joy. I only wish that more of them were on the service; I hope I've not lost all contact with those individuals.

Interestingly, participating in the Q&A section of the site has allowed me to establish connections to individuals I'd have never otherwise met or known. In particular, I'm now connected to someone in the Netherlands, someone in China, and have made the acquaintance of an individual in India. I have been able to provide career advice to a young professional searching for his way, and have been fortunate to receive many letters of thanks from both the question's authors and those who have found the words helpful.

The best thing LinkedIn has done for me, however, has nothing to do with their service. Realizing what LinkedIn really does happened in stages. There was the "What is this, really?" stage, then the "Wow, this could be a great showcase for me" stage, followed soon afterwards by the "Who else is on here" stage. Not long after that, a tiny dose of panic set in - how many people that I've known would be happy and eager to accept my invitation to connect? Any bit of self-doubt you had growing up in a cliquey school system comes rushing back. Fortunately, over 100 individuals I've known (and have recently met) are now connections; but the lesson taught by a desire to grow your network has been learned; Treat *everyone* you meet as if you'd like them to happily accept your invitation to connect.

Wednesday, April 16, 2008

Return of the Mac

A little over 12 months ago, I was fortunate enough to begin piloting an Apple MacBook Pro. It came with a 17" glossy display, 2GB of memory, pretty fast processor, and most importantly OS X Tiger.

After only a few weeks of use, I was so completely acclimated to the amazingly simple, elegant, efficient, effective design of the Mac that I began looking for ways to use them instead of Windows machines. Sadly, that effort stopped with Apple telling us they're unable to (and uninterested in) service broken systems on-site within any kind of SLA. This basically relegates Mac's to giant companies operating in a single campus, or companies with savvy IT hardware techs at each site - few of which are progressive enough to consider using Mac's.

After about 6 months, I started to forget how to use Windows as efficiently as I had before. What's funny about that fact is this; I held a job as a youngster which involved use of a character based point of sale system. I knew how to do everything with it, and could get around it with lightning speed - often helping my co-workers when they got stuck. Less than a year away from that job, I couldn't even tell you how to sign in anymore. It was completely gone.

The learning curve is so flat with Mac's, that you don't really have to "train" yourself to use them. It mostly involves forgetting many of the ill habits formed by using Windows. When you go back to Windows, you get frustrated very quickly at the stupid, work-losing default choices various buttons present.

Given that Gartner has recently declared Windows as "collapsing", my hope is that Apple gets over itself and creates an Apple Enterprise operating company focused on being a real alternative to the Big 3. I'm fairly certain that end-user software is slowly going to become OS independent, in one of a number of ways.

Nevertheless, I'd have been updating this blog more regularly were it not for Google's interference with the service. It simply would not let me log in using the credentials I made to create it. After some rapid (and unrepeatable) clicking, I finally found a screen that would let me re-enter the e-mail address. I will endeavor to update the blog with more frequency going forward.

Year in Review

Since it's been nearly a year between updates here, I should point out what's transpired.

Most significantly, I was promoted to Director of IS&T Infrastructure Services late last year. This essentially means I have full custody of the teams responsible for all desktop, server, and LAN/WAN environments - in addition to our support services. I hope eventually to back-fill my position, but much of that will depend on organizational issues above my pay grade (to borrow a term from some of my former military employees).

We're returning to Dell as a customer, after being so frustrated by the behemoth of IBM that we can simply no longer deal with them. To wit, we're replacing a 3 year old BladeCenter system with three Dell 2950's configured to run VMWare Infrastructure 3. We've also upgraded our Network Appliance FAS-270 to a FAS-3020, including the addition of 7TB of new disk - a process that was a bit more turbulent than we'd have hoped; vet your service providers carefully. That should give us plenty of capacity, and opens up a host of brilliant DR capabilities which we're excited to pursue this year.

We've also decided to put literally everything we do on the table for discussion. Decisions to either change or stay the course will need to be defensible, but everything is fair game. E-mail, file and print, networking services, etc. It's very exciting actually. Exiting this process, we'll have full and complete confidence that we're doing what's best for the business.

A more exciting development is the uncertainty surrounding some of the systems which have been outside my control. I'd have been far happier if we could have made the sale so to speak, and fully leveraged the investment in our ERP solution - the reality is that we didn't, and I think we're starting to face that fact by asking "how else can we tackle the problem"? It's a great opportunity to get much more closely involved with the business we support, and that type of experience is what I'm very much looking forward to in order to further my own professional development.

Finally, it was a great shame to see Martin Buckley leave Novell recently. I certainly do not fault him for doing so, and not just because I've made his acquaintance. Novell has deserved every bit of the intellectual capital loss they've experienced over the past few years. They are an organization which appears devoid of new ideas, leadership, and increasingly, relevance. The few bright spots within Provo are growing dimmer by the day. Plenty of blame to go around, for sure - Novell represents a master class in how not to run a technology company.