Lessons from a Botched Upgrade

[frommagio]

[ QUOTE ]
I think that's enough flaming for one thread. If you two must continue arguing, please take it to PMs.

[/ QUOTE ]

Ryan - You're right, and I appreciate that. I knew that my post would invite the inevitable 2+2 chatter, but I didn't want to let that prevent me from making a contribution of value.

I hope that you'll be able to use some of my ideas in planning/executing your next upgrade. Most of what I cited is pretty much standard industry practice for running online upgrades. It's true that 2+2 is not some mammoth Fortune 500 enterprise with a billion dollars, but I don't think that matters very much. Truth be told, you've actually got a big advantage - you're a small business with a fairly small upgrade problem, and you can be nimble. All you really need is the right mindset.

[frommagio]

[ QUOTE ]
fromm,

My problem is that the points in the OP are so wide as to be of limited use. What I suggest is that with your expertise, you could be gracious enough to offer more concrete suggestions that these guys can use.

[ QUOTE ]
Choose your technical staff wisely.

[/ QUOTE ]
Example: who should they be hiring to do this? Where should they be looking? Why?

[ QUOTE ]
- Test your upgrade first
-- Your customers are not a testbed.
-- Use a separate staging area.
Use artificial load generation to simulate real life.

[/ QUOTE ]
Briefly describe how this is done, who they'd hire, what setup they'd need, etc, to give them a feel for the process. Maybe provide websites with further information that you know from your profession.

If these guys are as green as you suggest then a couple of hundred words on this stuff could be very useful. As it is the OP is kind of vague.

[/ QUOTE ]

Phil -
It's true that many of the points were fairly wide, and they may seem a bit like motherhood. On the other hand, many of the fundamentals were clearly bypassed completely, so a bit of generalization and motherhood seems quite apropos, at least to me. The need for management to manage is really the key concept, with technology and technologists necessarily subservient. Note my claim that the root problem is managerial, not technological. Even in this technical world, the fundamental things are still very simple; e.g., the formulation of clear requirements, and the use of same in the review and sign-off process.

If the original post was too vague, I feel lucky in that I was able to flesh out some of the ideas in more detail in the follow-up posts, especially in the switchover plans. For the reader who cares to learn, there's some very solid information in there.

Naturally, I'll be happy to provide more when the time is right, if the 2+2 admins are interested and have questions. Right now, I think they are looking for a recovery period.

[ShakeZula06]

[ QUOTE ]
That's not always easy or viable. 2+2 isn't a multi billion dollar company.

[/ QUOTE ]
They are a multi million dollar company though, right?

Other then that I agree with most of Phil's points in his deleted post. OP is pretty much saying "If you're going to do something, do it right". That's all well and good, but it doesn't provide us with any lessons or insight.

[suzzer99]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]

Ryan - I don't mean to be rude, but

[/ QUOTE ]

No point reading after this, really.

You don't know their hardware, you don't know their budget, you don't know their requirements, you don't know what constraints they are operating under and you don't know their process. Good luck giving meaningful advice.

[/ QUOTE ]

We might not know the behind the scene details, but it is obvious that their is a problem with their current procedures. The most likely problem is that they are unable to replicate/simulate the amount of traffic that their production environment receives in their testing environment.


[/ QUOTE ]

There's load testing software out there that's not too expensive. Or you could take a couple hours and write a perl/VB/whatever script to simulate something roughly resembling normal traffic.

[KipBond]

[ QUOTE ]
[ QUOTE ]
If you needed to post a Friday night note that "service will be down until Sunday morning", you were already far off track.

[/ QUOTE ]

Why is that? The conversion of the db from 6.5 to 7 takes time. Originally it was going to take 3 days or more, but Chuck found ways to shorten that considerably.

[/ QUOTE ]

This leads me to an educated guess: the performance issues were probably db-related - most likely missing/corrupt indexes. But, regardless, use vBulletin next time. AND there should be very little downtime. A zero downtime plan takes a lot of planning, but you should be able to migrate to a new system in 3-4 hours downtime, max. Just do it in stages: a new parallel system with most of the current production data, down the production system, migrate the new data, bring up the new system.

[KipBond]

[ QUOTE ]
it would be very helpful if we can revisit all these topics as we get closer to said changes. i know that people have offered many helpful suggestions over the past year and many of them were lost in the shuffle.

i won't promise that our next attempts will be successful, but you can be damned sure we'll try harder. anyway. I 'm not trying to stop comments here. I just want everyone to be aware that any comments made now aren't being ignored, and when the process of upgrading the forums begins anew, please feel free to refresh our memory of today.

[/ QUOTE ]

With all due respect - seriously - honestly - I'm trying to be very nice here - .... please take notes; write down the good suggestions, and discuss them with those that know what they're doing.

[KipBond]

[ QUOTE ]
The key to getting on track is to realize that management mistakes are the issue.

[/ QUOTE ]

Specifically, IT management. But I don't know their staff organizational structure or budget -- so it's hard to say how easy this will be to fix. Some places it's really hard to get the right person to do the job (political and financial reasons, for example).

[bav]

[ QUOTE ]
[ QUOTE ]
The key to getting on track is to realize that management mistakes are the issue.

[/ QUOTE ]

Specifically, IT management. But I don't know their staff organizational structure or budget -- so it's hard to say how easy this will be to fix. Some places it's really hard to get the right person to do the job (political and financial reasons, for example).

[/ QUOTE ]
I mostly agree with all the stuff magio and related agreers have posted. But it has to be tempered by the details of the 2+2 business as stated above.

Yeah, a professional team of people with an experienced IT manager at a large corporation probably woulda done things differently. Yahoo, AOL, MSN, Google, etc do NOT go down to do upgrades. The changes either just happen POOF or at most there's a "oops--your login session just dropped" sorta discontinuity. 17 hours of outtage 10 years ago got AOL front page headlines all over the planet and expectations are much higher today.

But 2+2 isn't AOL and the reliability and availability requirements aren't the same. It really kinda is ok for 2+2 to go down for 15 minutes. It'd be better if it didn't, but... 99.9% is probably enough for most web sites and the cost of an extra couple 9's can be pretty high. But I wouldn't consider it ok for 2+2 to be down for 24 hours if I were in charge. And the chatter about their IT management and organizational structure probably is worthy of giggles; I just don't see 2+2 having a Director of IT with 3 managers reporting to him and each manager with a staff of 12. I get the feeling it's more a couple guys in their home offices.

The tech folks SHOULD be able to propose a plan to do a 0-downtime upgrade. It likely will require having a whole separate system set up and running as magio described, but doing that means it should have been thoroughly beta tested and load tested and with just a brief outage it could be cut in. And if something goes wrong, just as quickly yanked. If they want to pay for it.

If the management of 2+2 comes back saying "can you save some money", it's incumbent on the tech folks to come up with another, cheaper plan. And spell out the increased risks and impact on availability.

But I think the default should be a minimal downtime, minimal risk approach. Only when that is prohibitively slow or expensive should one think about multi-day outages, and always there needs to be the backout plan. I don't know how long 2+2 was completely down and out for the conversion, but even after it came back up it was effectively unusable. I'm quite baffled as to why it was allowed to stay like that for so long. When you do an upgrade, you have a backout plan. When the upgrade isn't working out as planned, you don't just leave it there for two days and hope some miracle will happen.

So yeah, I think this was a botched upgrade and yeah, there are lessons to be learned, but I sure don't know enough about how it was planned and executed and who made the decisions to know where to point a finger. But I do think 2+2 is too large and successful to treat it like a high school senior class web project.

[KipBond]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
The key to getting on track is to realize that management mistakes are the issue.

[/ QUOTE ]

Specifically, IT management. But I don't know their staff organizational structure or budget -- so it's hard to say how easy this will be to fix.

[/ QUOTE ]
And the chatter about their IT management and organizational structure probably is worthy of giggles; I just don't see 2+2 having a Director of IT with 3 managers reporting to him and each manager with a staff of 12. I get the feeling it's more a couple guys in their home offices.

[/ QUOTE ]

Yeah, I get the same feeling. Part time job - maybe even volunteer? I'm coming from the POV of being in a small organization with a single IT guy (me) that does everything. I would think that 2+2 could afford to get some contract labor to help them out with this migration. I could be wrong, and it might be more than money issues.