There is a gigantic field of people who are chomping at the bit to do SharePoint consulting. They all seem to be very busy, as well. Most of them have polished presentations and speak intelligently about their methodology, the importance of interviewing employees and taking time to document process flows, document types, develop taxonomies, etc.
None of them, at least that we've seen, are very good at running a project from start to finish if you don't have endless years and deep pockets.
If you don't have an appetite for custom development, opting instead to use that which you can support yourself absent an army of SharePoint coders, getting SharePoint installed becomes a very trying exercise.
There are some things that SharePoint does well, and that will be valuable to enterprise organizations who invest in their IT groups (rather than just spending money on them, which is a big difference).
However, most of those things - from an administrative standpoint; from the perspective of an IT director; are overshadowed by all of the things SharePoint either does poorly, or not at all. Even in MOSS 2007, even with the enterprise version of the product, there's a lot that can (and does) frustrate a quality-minded IT manager.
It seems like we might have been the last people on earth to embark upon a SharePoint deployment, but at times, it seemed like we might have been the first. Our knack for running into legitimate issues and bugs that "nobody" has seen before is truly miraculous - a candidate for a well funded study if ever there was one.
If you have NOT yet drank the MOSS kool-aide, and do not have an endless supply of developers to whom you can throw every little requirement, keep the following in mind: implementing it is going to be a problem, but nothing you cannot overcome. It will fall short of your expectations. It will not go smoothly. You will start to become adept at finding the capability gaps in the people delivering the solution to you, and will realize what we have - that NOBODY is good at this.
Maybe it'll be better in the next version.
Tuesday, November 24, 2009
Thursday, October 08, 2009
Still with the Cold Calling? Really?
I'm baffled that this actually works, but it must, because there are so many companies doing it. I realize this is the third time in as many years that I've written about this, but it is endlessly frustrating yet amusing in a "Curb Your Enthusiasm" sort of way.
For those unfamiliar with the principle, get this. People will actually call up out of the blue, ask to speak to "the IT department", and announce quickly that "they're not selling anything" (because that would be unspeakable!), but rather, they're collecting answers on topic X, Y, or Z for some anonymous cabal of vendors too inept to figure out what the market wants or needs.
Our favorite response is to let the caller know that we have corporate rules restricting us from answering survey questions. Plausible deniability. Sometimes it gets you off of the list for good, sometimes not, but it always makes for a short call.
Just as baffling are the people that survey callers look down upon - the cold calling salesperson. Literally phone a company at random, as for someone, and try to convince them that they should give you the time of day (much less start spending money with you) in the span of about 15 seconds.
Who are the people that keep feeding the birds? Don't feed the birds. Didn't your parents teach you anything? You feed the birds, and more and more show up, and it ruins things for everybody. Don't feed the birds. Do NOT respond to cold callers.
Here's how conversations go for me with these people lately.
The long form:
(phone rings)
ZM: "Zen Master here."
CC: "Hi, can I speak to the person in charge of purchasing decisions for X?" (usually toner or backup tapes or cell phone accessories or something else stupid)
ZM: "That's me, what can we do for you?"
CC: "I'm with Bob's Pretty Good company, have you heard of us?"
ZM: "No."
CC: "Well I'd like to see if we can get on your schedule to talk about all the ways we can save your company money and make everyone more efficient and clear up bad complexions and cure swine flu...what does your calendar look like?"
ZM: "We're all set on that front, but I appreciate the call."
CC: "So what do you use for swine flu and bad complexions?"
ZM: "I'm not inclined to reveal that to you." (click)
The short form:
(phone rings)
ZM: "This is the Zen master."
CC: "Hi Zen master, I'm with Radioactive Toner company and wanted to see if you had any upcoming projects, you know, involving toner, that we might be able to help with."
ZM: "We're all set on toner, but thanks for the call." (click)
This one is interesting too, ever since we became EA customers, people are presenting themselves as being with Microsoft. A particularly sneaky tactic, MS will actually badge channel partners as employees, so they can 'legitimately' say they are with Microsoft. I have nothing but loathing for this form of deceit. I am shocked, SHOCKED, that my influential blog has not put a stop to this practice once and for all.
The liar:
(phone rings)
ZM: "Zen master here."
CC: "Is this the Zen master?"
ZM: "Yes, who is this?"
CC: "I'm Bob calling on behalf of Microsoft."
ZM: "Who are you with?"
CC: "Microsoft?"
ZM: "You aren't on my list of account representatives."
CC: "Well I'm calling on behalf of Microsoft."
ZM: "That's why I asked who you work for."
CC: "Oh, I'm with (some company nobody ever heard of)."
ZM: "Okay, thank you." (click)
One guy recently never came clean, and I knew it, but acted like I didn't. Genuine account reps know everything about us...if someone calling doesn't, they are not with Microsoft.
Perhaps one of you will know what website I can go to in order to get a rebate on all the time I waste with these people. If not, hopefully you get a little satisfaction knowing you're not alone (or that your job could be a lot worse).
For those unfamiliar with the principle, get this. People will actually call up out of the blue, ask to speak to "the IT department", and announce quickly that "they're not selling anything" (because that would be unspeakable!), but rather, they're collecting answers on topic X, Y, or Z for some anonymous cabal of vendors too inept to figure out what the market wants or needs.
Our favorite response is to let the caller know that we have corporate rules restricting us from answering survey questions. Plausible deniability. Sometimes it gets you off of the list for good, sometimes not, but it always makes for a short call.
Just as baffling are the people that survey callers look down upon - the cold calling salesperson. Literally phone a company at random, as for someone, and try to convince them that they should give you the time of day (much less start spending money with you) in the span of about 15 seconds.
Who are the people that keep feeding the birds? Don't feed the birds. Didn't your parents teach you anything? You feed the birds, and more and more show up, and it ruins things for everybody. Don't feed the birds. Do NOT respond to cold callers.
Here's how conversations go for me with these people lately.
The long form:
(phone rings)
ZM: "Zen Master here."
CC: "Hi, can I speak to the person in charge of purchasing decisions for X?" (usually toner or backup tapes or cell phone accessories or something else stupid)
ZM: "That's me, what can we do for you?"
CC: "I'm with Bob's Pretty Good company, have you heard of us?"
ZM: "No."
CC: "Well I'd like to see if we can get on your schedule to talk about all the ways we can save your company money and make everyone more efficient and clear up bad complexions and cure swine flu...what does your calendar look like?"
ZM: "We're all set on that front, but I appreciate the call."
CC: "So what do you use for swine flu and bad complexions?"
ZM: "I'm not inclined to reveal that to you." (click)
The short form:
(phone rings)
ZM: "This is the Zen master."
CC: "Hi Zen master, I'm with Radioactive Toner company and wanted to see if you had any upcoming projects, you know, involving toner, that we might be able to help with."
ZM: "We're all set on toner, but thanks for the call." (click)
This one is interesting too, ever since we became EA customers, people are presenting themselves as being with Microsoft. A particularly sneaky tactic, MS will actually badge channel partners as employees, so they can 'legitimately' say they are with Microsoft. I have nothing but loathing for this form of deceit. I am shocked, SHOCKED, that my influential blog has not put a stop to this practice once and for all.
The liar:
(phone rings)
ZM: "Zen master here."
CC: "Is this the Zen master?"
ZM: "Yes, who is this?"
CC: "I'm Bob calling on behalf of Microsoft."
ZM: "Who are you with?"
CC: "Microsoft?"
ZM: "You aren't on my list of account representatives."
CC: "Well I'm calling on behalf of Microsoft."
ZM: "That's why I asked who you work for."
CC: "Oh, I'm with (some company nobody ever heard of)."
ZM: "Okay, thank you." (click)
One guy recently never came clean, and I knew it, but acted like I didn't. Genuine account reps know everything about us...if someone calling doesn't, they are not with Microsoft.
Perhaps one of you will know what website I can go to in order to get a rebate on all the time I waste with these people. If not, hopefully you get a little satisfaction knowing you're not alone (or that your job could be a lot worse).
Friday, August 21, 2009
Progress At Last
We finally got the issue with our image resolved and were able to deploy all of our new hardware...just under 500 systems, in just over 3 weeks in June (we took it easy). Everyone seems very happy with them thus far...lots faster, for certain. I do wish the connection manager component with Dell was as good as IBM's Thinkvantage Access Connections, so we'll keep pushing for improvements.
Finally getting some traction on the SharePoint front as well, using a consulting "mercenary" of sorts who has done a great job helping us map our requirements to features, and actually show us what things will look like as we go.
Windows Server 2008 R2 looks as if it may have some interesting benefits for us, as it applies to the total mobility effort we're undertaking. What we have in the lab now isn't far off, frankly, and it's pretty exciting to see it come together. 2008 R2 makes a compelling case to use Windows 7 once it's finally released also, which is depressing in a way, but is something we'll have to keep an eye on.
We just renewed our EA and Support agreements with Microsoft. I had high expectations but low hopes that our Microsoft relationship would be anything other than artificially pleasant and devoid of substance. In short, I've never been so happy to be wrong in my life. What a great lesson.
Finally getting some traction on the SharePoint front as well, using a consulting "mercenary" of sorts who has done a great job helping us map our requirements to features, and actually show us what things will look like as we go.
Windows Server 2008 R2 looks as if it may have some interesting benefits for us, as it applies to the total mobility effort we're undertaking. What we have in the lab now isn't far off, frankly, and it's pretty exciting to see it come together. 2008 R2 makes a compelling case to use Windows 7 once it's finally released also, which is depressing in a way, but is something we'll have to keep an eye on.
We just renewed our EA and Support agreements with Microsoft. I had high expectations but low hopes that our Microsoft relationship would be anything other than artificially pleasant and devoid of substance. In short, I've never been so happy to be wrong in my life. What a great lesson.
Friday, April 17, 2009
Sea Change
While we still don't have our imaging show-stopper taken care of, we're a lot closer. In fact, we're about to download the latest version of our image on ISO's to test, and if the 'fix' we've received does the trick, we'll finally be able to deploy our desktop and laptop hardware - about three months late.
On another front, we were able to migrate off of Novell GroupWise and on to Exchange 2007, with surprisingly little fuss. We used ZENworks to deploy Office 2007 silently, and with all of our Outlook settings pre-populated. We obviously deployed Microsoft Active Directory, using Server 2008 and native mode (I think).
Finally, we dropped our Novell licenses in favor of renewing only the ZEN 10 products for configuration management, asset management, and patch management. This allows us to bring forward the ZEN functionality we were unable to find anywhere else, while finally ridding ourselves of this increasingly cumbersome Novell infrastructure.
We're looking forward to taking full advantage of MS Server 2008 features such as DFS, to simplify backups, file & printer sharing, etc. at our remote sites. We'll be looking closely at Cisco WAAS products in this space.
We've also integrated our network with that of a recent acquisition, using an internet-based VAS Half Tunnel from Sprint. The connection between our networks is 10Mbps, which performs very well so far despite lack of end-to-end CoS/QoS tagging (call it what you want). The next steps there are to basically import their users and e-mail into our AD and Exchange system, using local servers, and begin doing some cool DR stuff using VMware and (hopefully) NetApp storage at the far end.
Despite the bad economy, we're busier than ever. We're nearly finished implementing Microsoft SharePoint (MOSS 2007), and are about to implement Microsoft OCS as well.
The complexion of this environment has undergone rapid, radical change, and we've been successful at facilitating this change with a very small but talented group of individuals. It took around 1,200 man hours over a month to get Exchange in place, and we completed the migration in less than 72 hours. After some post-implementation firefighting, we're pretty confident that we didn't lose a single mailbox, address book, or archive.
Our satisfaction survey following the Outlook/Exchange migration was overwhelmingly positive. In fact, only two of our roughly 500 employees had nothing positive to say. Many went out of their way to complement us on the process, and the decision to do so. Happy customers are a good thing.
On another front, we were able to migrate off of Novell GroupWise and on to Exchange 2007, with surprisingly little fuss. We used ZENworks to deploy Office 2007 silently, and with all of our Outlook settings pre-populated. We obviously deployed Microsoft Active Directory, using Server 2008 and native mode (I think).
Finally, we dropped our Novell licenses in favor of renewing only the ZEN 10 products for configuration management, asset management, and patch management. This allows us to bring forward the ZEN functionality we were unable to find anywhere else, while finally ridding ourselves of this increasingly cumbersome Novell infrastructure.
We're looking forward to taking full advantage of MS Server 2008 features such as DFS, to simplify backups, file & printer sharing, etc. at our remote sites. We'll be looking closely at Cisco WAAS products in this space.
We've also integrated our network with that of a recent acquisition, using an internet-based VAS Half Tunnel from Sprint. The connection between our networks is 10Mbps, which performs very well so far despite lack of end-to-end CoS/QoS tagging (call it what you want). The next steps there are to basically import their users and e-mail into our AD and Exchange system, using local servers, and begin doing some cool DR stuff using VMware and (hopefully) NetApp storage at the far end.
Despite the bad economy, we're busier than ever. We're nearly finished implementing Microsoft SharePoint (MOSS 2007), and are about to implement Microsoft OCS as well.
The complexion of this environment has undergone rapid, radical change, and we've been successful at facilitating this change with a very small but talented group of individuals. It took around 1,200 man hours over a month to get Exchange in place, and we completed the migration in less than 72 hours. After some post-implementation firefighting, we're pretty confident that we didn't lose a single mailbox, address book, or archive.
Our satisfaction survey following the Outlook/Exchange migration was overwhelmingly positive. In fact, only two of our roughly 500 employees had nothing positive to say. Many went out of their way to complement us on the process, and the decision to do so. Happy customers are a good thing.
Monday, February 02, 2009
The Age Of Careless Development
It's become really quite alarming lately, just how bad software has become. Everywhere. It's impossible to single out a given vendor for being worse than the rest, without being entirely subjective. They all stink, equally, but in different ways.
In fact, I cannot think of a single software vendor at the moment - or even hardware vendors who write complementary software packages - who are anywhere near proficient or at least competent.
I mentioned to a vendor earlier today, that "we" (as in, the IT industry) is worse at managing computer hardware now than we were 6 years ago. Dell has made the ImageDirect process unbelievably cumbersome. Microsoft's SMS / SCCM is still unweildy and ineffective, ZENworks (while still one of the better offerings) has launched down a rabbit hole so deep and dark that nobody dare follow - to say nothing of the fact that their company isn't interesting to anyone anymore (vis-a-vis the cancellation of BrainShare for the first time ever). Lenovo just flat gave up on imaging, pushing the task to "partners" like Insight, who as box pushers, struggle to manage the lifecycle of a product-customer relationship well at all.
Referring to our current image-impacting show-stopping software bug, that vendor said "Man, I hope desktop virtualization fixes this." And so do I, but I'm still skeptical.
Somewhere along the line, the stuff that made great programmers and resulted in great software, has become lost.
It's almost as if the tools became too easy. Anyone could make software that looked real, because in Windows-land, it all basically looks the same. What's masked is the hard and careful work that used to go into making it work well - which was usually far more important than how it looked. Some of the best running software looked the worst. Take RPG applications on AS/400. They. Just. Never. Fail.
Programmers used to be a unique breed, because not everyone could visualize how everything worked, and make it do what they wanted. Nowadays, you just have to read some Visual Crap++ books or watch a webinar, and viola - you're a developer.
Software developers need to be reconnected with those of us who USE their code day-in and day-out. The whiteboard upon which requirements were drawn will never use the software they produce - people do. If you've had to manage people, or ever managed an IT group, you can spot fundamental, architectural problems in software from a country mile. But developers these days don't have that experience, and they don't get that guidance from their product managers. Which means they churn out code that met requirements only the straw man from "Wizard of Oz" could ever use.
Making matters worse is when companies give up on actual development, farming it off to "code vendors" in favor of becoming keepers of the Gantt chart instead. Development is an organic process, and the people who are the best at it are the ones who are genuinely invested in the success of the organization for whom they're doing development. When the guy writing the code is never going to talk to someone using it (or even the person buying it), they're a lot less accountable - which means they're a lot less careful, or proactive, or focused.
These are all easy problems to solve, and if we're truly entering the age of the cloud where software isn't the commodity anymore, it's absolutely crucial that companies recognize the advantages of experience-based development. Rational Rose, rapid, lean, whatever the buzzword is today, are no substitutes for development lifecycles that involve seasoned professionals who know how EVERYTHING works together.
In fact, I cannot think of a single software vendor at the moment - or even hardware vendors who write complementary software packages - who are anywhere near proficient or at least competent.
I mentioned to a vendor earlier today, that "we" (as in, the IT industry) is worse at managing computer hardware now than we were 6 years ago. Dell has made the ImageDirect process unbelievably cumbersome. Microsoft's SMS / SCCM is still unweildy and ineffective, ZENworks (while still one of the better offerings) has launched down a rabbit hole so deep and dark that nobody dare follow - to say nothing of the fact that their company isn't interesting to anyone anymore (vis-a-vis the cancellation of BrainShare for the first time ever). Lenovo just flat gave up on imaging, pushing the task to "partners" like Insight, who as box pushers, struggle to manage the lifecycle of a product-customer relationship well at all.
Referring to our current image-impacting show-stopping software bug, that vendor said "Man, I hope desktop virtualization fixes this." And so do I, but I'm still skeptical.
Somewhere along the line, the stuff that made great programmers and resulted in great software, has become lost.
It's almost as if the tools became too easy. Anyone could make software that looked real, because in Windows-land, it all basically looks the same. What's masked is the hard and careful work that used to go into making it work well - which was usually far more important than how it looked. Some of the best running software looked the worst. Take RPG applications on AS/400. They. Just. Never. Fail.
Programmers used to be a unique breed, because not everyone could visualize how everything worked, and make it do what they wanted. Nowadays, you just have to read some Visual Crap++ books or watch a webinar, and viola - you're a developer.
Software developers need to be reconnected with those of us who USE their code day-in and day-out. The whiteboard upon which requirements were drawn will never use the software they produce - people do. If you've had to manage people, or ever managed an IT group, you can spot fundamental, architectural problems in software from a country mile. But developers these days don't have that experience, and they don't get that guidance from their product managers. Which means they churn out code that met requirements only the straw man from "Wizard of Oz" could ever use.
Making matters worse is when companies give up on actual development, farming it off to "code vendors" in favor of becoming keepers of the Gantt chart instead. Development is an organic process, and the people who are the best at it are the ones who are genuinely invested in the success of the organization for whom they're doing development. When the guy writing the code is never going to talk to someone using it (or even the person buying it), they're a lot less accountable - which means they're a lot less careful, or proactive, or focused.
These are all easy problems to solve, and if we're truly entering the age of the cloud where software isn't the commodity anymore, it's absolutely crucial that companies recognize the advantages of experience-based development. Rational Rose, rapid, lean, whatever the buzzword is today, are no substitutes for development lifecycles that involve seasoned professionals who know how EVERYTHING works together.
Wednesday, October 29, 2008
Novell Sees The Light on Maintenance
We processed our renewal today, and learned that Novell is now including unlimited support with the software maintenance MLA customers purchase each year. This prevents you from needing to buy Premium 1000 or 2000 (or more), and be worrying about how many times you call.
This is much more consistent with enterprise vendors like Oracle, IBM, and Sun, and is a welcome change. Unfortunately, it's a case of too little, too late. We're done waiting for Novell to do what customers like us have told them for 4, 5 or 6 years. As the saying goes down here, "Even a good dog will only stay on the porch for so long."
I'm told that more changes are coming over the next year, and I'm definitely interested in hearing what they are. I suspect it will address many of the complaints and confusing practices we've called out here, leaving only the question of "What took you so long?" for them to answer.
This is much more consistent with enterprise vendors like Oracle, IBM, and Sun, and is a welcome change. Unfortunately, it's a case of too little, too late. We're done waiting for Novell to do what customers like us have told them for 4, 5 or 6 years. As the saying goes down here, "Even a good dog will only stay on the porch for so long."
I'm told that more changes are coming over the next year, and I'm definitely interested in hearing what they are. I suspect it will address many of the complaints and confusing practices we've called out here, leaving only the question of "What took you so long?" for them to answer.
Tuesday, October 14, 2008
OES Linux and 8 Character Passwords
Did you know that OES on SLES is incapable of using passwords for the Root user longer than 8 characters?
...
Did you also know that OES on SLES allows you to blank out the Root user password using the passwd utility?
...
Did you know that lots of stuff is dependent on the Root user, and that password cannot be controlled through NAMCD or eDirectory?
...
Hoo boy.
...
Did you also know that OES on SLES allows you to blank out the Root user password using the passwd utility?
...
Did you know that lots of stuff is dependent on the Root user, and that password cannot be controlled through NAMCD or eDirectory?
...
Hoo boy.
Friday, September 12, 2008
Here It Comes (Microsoft Update)
Well, so much for saving money using Microsoft versus Novell. :-)
If ALL we wanted to do was apples-to-apples functionality, and had no needs for the more advanced (and to us, compelling) SharePoint features, we'd probably be able to do more for about the same spend.
But we DO need those Enterprise features, and that means the E-CAL suite. And that means we're definitely spending more than we used to.
What a lot of antagonists of Microsoft don't understand, for all their yelling and fear-mongering, is that the money you need to run Microsoft products is still barely a blip for big companies.
Novell then, as victims of their own success, have relegated themselves to the small-business/school/government niches where they've always done well - places where price trumps value.
Where companies make decisions based on value - not just price - Microsoft wins, hands down. Even if it's more expensive than Novell, it's still cheap. I can run our enterprise on about $70,000 worth of Novell licensing a year, including their pathetic excuse for phone support. Our Microsoft bill is likely to be around 4 times that, but that includes Office Pro Plus, and the entire E-CAL suite. Compare that to what a company spends per-head on enterprise systems like Oracle, etc., and it's a drop in the bucket. More people will get more productivity and efficiency out of MS products at that price than they ever will from their ERP system.
Anyway, we're looking forward to getting up to speed on the MS technologies - we're having in-house training performed, tailored to our specs and individuals, by an MS certified education partner. Other organizations here in town have done the same with this company, and have been very pleased.
If only Dell would release their E-Series laptop, we could really get moving with this effort.
If ALL we wanted to do was apples-to-apples functionality, and had no needs for the more advanced (and to us, compelling) SharePoint features, we'd probably be able to do more for about the same spend.
But we DO need those Enterprise features, and that means the E-CAL suite. And that means we're definitely spending more than we used to.
What a lot of antagonists of Microsoft don't understand, for all their yelling and fear-mongering, is that the money you need to run Microsoft products is still barely a blip for big companies.
Novell then, as victims of their own success, have relegated themselves to the small-business/school/government niches where they've always done well - places where price trumps value.
Where companies make decisions based on value - not just price - Microsoft wins, hands down. Even if it's more expensive than Novell, it's still cheap. I can run our enterprise on about $70,000 worth of Novell licensing a year, including their pathetic excuse for phone support. Our Microsoft bill is likely to be around 4 times that, but that includes Office Pro Plus, and the entire E-CAL suite. Compare that to what a company spends per-head on enterprise systems like Oracle, etc., and it's a drop in the bucket. More people will get more productivity and efficiency out of MS products at that price than they ever will from their ERP system.
Anyway, we're looking forward to getting up to speed on the MS technologies - we're having in-house training performed, tailored to our specs and individuals, by an MS certified education partner. Other organizations here in town have done the same with this company, and have been very pleased.
If only Dell would release their E-Series laptop, we could really get moving with this effort.
Friday, August 22, 2008
VMWare and NFS on NetApp Filers
Ran across a very disturbing issue this week, which caused the corruption of nearly half of our virtual machines - certainly all of the ones which were experiencing any load to speak of.
If you run VMWare ESX 3i against a NetApp filer using storage pools defined as NFS mounts (of which a given cluster can only have 32, by the way), and you're a good Systems Engineer and follow the NetApp best practices guidelines for configuring ESX to work optimally with their products, then you'll turn on this nifty little switch called NFS.Lock.Disable by changing the default value of "0" to "1".
The document in question from October, 2007, used to be on the NetApp site (and maybe VMware's as well), but is now relegated to one of those cool sites that leech stuff from the web and hold it forever.
http://www.docstoc.com/docs/304765/Network-Appliance-and-VMware-Virtual-Infrastructure-3-Storage-Best-Practices
Then about 10 months later, or 2 months ago, it changed.
http://media.netapp.com/documents/tr-3428.pdf
Suddenly, no mention of NFS.Lock.Disable.
This document, on the VMware website, however interestingly titled, makes no mention of it the setting. It's authored by NetApp as well, and is dated May 2007.
http://www.vmware.com/files/pdf/partners/netapp_esx_best_practices_whitepaper.pdf
So what gives? Why so many documents and versions on the same topic, by so many authors? Who knows. All we know is that "best practices" for VMWare ESX 3 customers using NetApp and NFS storage pools was to set NFS.Lock.Disable to 1.
The problem is that VMWare ESX 3i isn't what ESX 3 used to be. From the poor retention of logs (not my opinion; statement of fact from VMware support), to the singular dependency on file locks to ensure split brain conditions don't occur, all the way to patches that somehow slip past QC with license-detonation code included.
All of this means when you do what VMware and NetApp told you to do in order to deploy their products together successfully, you put your data at risk.
And we did, and ours was, and it sucked.
Split brain means, in essence, that the system in charge of deciding what VM goes where (VCenter in this case) is unaware or uncertain of what ESX host holds which guest operating systems. In this condition, two separate ESX hosts can be - simultaneously - running the exact same guest OS instances. The bad news is, as you might imagine, that one half of that "brain" doesn't know what the other half is doing. What results is the utter annihilation of your filesystem, and depending on where and how you keep files, a very long process of restoring to a known good state.
The symptoms are beyond bizarre. VCenter shows a guest VM as being on a different host just about every second. Opening up a VM's console may give two people completely different screens, because you're actually looking at different "real" instances of the same virtual machine. Shutting down VCenter doesn't make things better; connecting directly to the ESX host will show you a wildly fluctuating number of guests running.
The only remedy is to use some neat "unsupported" (wink wink) console commands on the ESX hosts, and 'kill' the offending VM's. The faster you do that, the less badly your data will be corrupted....no matter how you slice it, it stinks.
VMware's Platinum support was surprisingly disappointing. Our first tech "hadn't been trained on 3i yet", and it took a while to get to someone who was. Like, hours. The RCLI is lacking commands that ESX 3 used to have, which vexes them. Following a host reboot, logs aren't kept. At all. They're not kept at all. What? Yes. Used to be in 3, but not in 3i. Nice. Once we got back up, it took ages and a lot of pretty irate e-mail to get someone to do some post-mortem analysis. Ultimately, we heard the details here straight from the horse's mouth. They're trying to eradicate the versions of that document that advise NFS.Lock.Disable - word never made it to us, somehow. They say they've known about the problem for about 30 days, which seems unrealistic. They say about 12 customers have had the same exact issues. And, unequivocally, they say to turn off that NFS.Lock.Disable shit, post haste.
My hope is that people who have deployed with this setting seriously consider changing it, and perhaps ask NetApp WTF? (or better yet, share their experiences in the comment area below). And additionally, my hope is that we're able to convince VMware that the deficiencies with 3i are totally unacceptable (no matter how insignificant they may seem to those with direct-attached or FC-attached disk). They could take a cue from Novell regarding heartbeat and keepalives to make sure direct host-to-host communication is used as a failsafe against these goofy file locks before allowing a VM to start on a new host.
If you run VMWare ESX 3i against a NetApp filer using storage pools defined as NFS mounts (of which a given cluster can only have 32, by the way), and you're a good Systems Engineer and follow the NetApp best practices guidelines for configuring ESX to work optimally with their products, then you'll turn on this nifty little switch called NFS.Lock.Disable by changing the default value of "0" to "1".
The document in question from October, 2007, used to be on the NetApp site (and maybe VMware's as well), but is now relegated to one of those cool sites that leech stuff from the web and hold it forever.
http://www.docstoc.com/docs/304765/Network-Appliance-and-VMware-Virtual-Infrastructure-3-Storage-Best-Practices
Then about 10 months later, or 2 months ago, it changed.
http://media.netapp.com/documents/tr-3428.pdf
Suddenly, no mention of NFS.Lock.Disable.
This document, on the VMware website, however interestingly titled, makes no mention of it the setting. It's authored by NetApp as well, and is dated May 2007.
http://www.vmware.com/files/pdf/partners/netapp_esx_best_practices_whitepaper.pdf
So what gives? Why so many documents and versions on the same topic, by so many authors? Who knows. All we know is that "best practices" for VMWare ESX 3 customers using NetApp and NFS storage pools was to set NFS.Lock.Disable to 1.
The problem is that VMWare ESX 3i isn't what ESX 3 used to be. From the poor retention of logs (not my opinion; statement of fact from VMware support), to the singular dependency on file locks to ensure split brain conditions don't occur, all the way to patches that somehow slip past QC with license-detonation code included.
All of this means when you do what VMware and NetApp told you to do in order to deploy their products together successfully, you put your data at risk.
And we did, and ours was, and it sucked.
Split brain means, in essence, that the system in charge of deciding what VM goes where (VCenter in this case) is unaware or uncertain of what ESX host holds which guest operating systems. In this condition, two separate ESX hosts can be - simultaneously - running the exact same guest OS instances. The bad news is, as you might imagine, that one half of that "brain" doesn't know what the other half is doing. What results is the utter annihilation of your filesystem, and depending on where and how you keep files, a very long process of restoring to a known good state.
The symptoms are beyond bizarre. VCenter shows a guest VM as being on a different host just about every second. Opening up a VM's console may give two people completely different screens, because you're actually looking at different "real" instances of the same virtual machine. Shutting down VCenter doesn't make things better; connecting directly to the ESX host will show you a wildly fluctuating number of guests running.
The only remedy is to use some neat "unsupported" (wink wink) console commands on the ESX hosts, and 'kill' the offending VM's. The faster you do that, the less badly your data will be corrupted....no matter how you slice it, it stinks.
VMware's Platinum support was surprisingly disappointing. Our first tech "hadn't been trained on 3i yet", and it took a while to get to someone who was. Like, hours. The RCLI is lacking commands that ESX 3 used to have, which vexes them. Following a host reboot, logs aren't kept. At all. They're not kept at all. What? Yes. Used to be in 3, but not in 3i. Nice. Once we got back up, it took ages and a lot of pretty irate e-mail to get someone to do some post-mortem analysis. Ultimately, we heard the details here straight from the horse's mouth. They're trying to eradicate the versions of that document that advise NFS.Lock.Disable - word never made it to us, somehow. They say they've known about the problem for about 30 days, which seems unrealistic. They say about 12 customers have had the same exact issues. And, unequivocally, they say to turn off that NFS.Lock.Disable shit, post haste.
My hope is that people who have deployed with this setting seriously consider changing it, and perhaps ask NetApp WTF? (or better yet, share their experiences in the comment area below). And additionally, my hope is that we're able to convince VMware that the deficiencies with 3i are totally unacceptable (no matter how insignificant they may seem to those with direct-attached or FC-attached disk). They could take a cue from Novell regarding heartbeat and keepalives to make sure direct host-to-host communication is used as a failsafe against these goofy file locks before allowing a VM to start on a new host.
Tuesday, July 01, 2008
Where Apple Won't Go
I remain befuddled by Apple Computer's outright thickheadedness with regard to their products and the American enterprise.
It's plain for all to see that Apple want absolutely no part of selling into corporations other than boutique art houses, or any other non-consumer segment other than schools. Yet despite this, Apple publishes a link to a story - on their own Apple "Start" page no less - highlighting the fact that 8 in 10 companies in America have Mac computers in production.
...
So, which is it - you don't care about corporations, or you're proud of the penetration your products have made into the enterprise space?
If it's the latter, how about actually ramping up an enterprise business unit? You know, with financing and on-site support programs and all the rest, like the other big kids?
Apple has done so poorly at speaking to the enterprise, that a group of five companies this week decided they'd do it for them. I love one of the quotes in the story.
That's for damn sure.
Given that the newest iPhone is touted as having "enterprise" hooks, vis-a-vis integration of sorts with Exchange mail systems (not exactly in-use at many homes), one can hope that Apple's efforts to get into the enterprise will become a bit more purposeful than the X Server or X San. They need only the slightest breath to break the logjam holding Mac computers back from overrunning the enterprise space. That they pay us no attention is becoming less of an interesting quirk, and more of an insult.
It's plain for all to see that Apple want absolutely no part of selling into corporations other than boutique art houses, or any other non-consumer segment other than schools. Yet despite this, Apple publishes a link to a story - on their own Apple "Start" page no less - highlighting the fact that 8 in 10 companies in America have Mac computers in production.
...
So, which is it - you don't care about corporations, or you're proud of the penetration your products have made into the enterprise space?
If it's the latter, how about actually ramping up an enterprise business unit? You know, with financing and on-site support programs and all the rest, like the other big kids?
Apple has done so poorly at speaking to the enterprise, that a group of five companies this week decided they'd do it for them. I love one of the quotes in the story.
"There's an information vacuum that we want to fill," T. Reid Lewis, president of Group Logic, told InformationWeek.
That's for damn sure.
Given that the newest iPhone is touted as having "enterprise" hooks, vis-a-vis integration of sorts with Exchange mail systems (not exactly in-use at many homes), one can hope that Apple's efforts to get into the enterprise will become a bit more purposeful than the X Server or X San. They need only the slightest breath to break the logjam holding Mac computers back from overrunning the enterprise space. That they pay us no attention is becoming less of an interesting quirk, and more of an insult.
Subscribe to:
Posts (Atom)