16 posts tagged “nightmares”
On Saturday morning (yes, I work 6 days a week now), I got this e-mail (replicated here in full) from the project's director, in response to an e-mail I sent out detailing some issues with our project:
We need to get the issues in Train resolved first. Who is working on these issues from the dev team?
Excuse me? Who's the director here? Me or you? I suppose the question could have been directed at his counterpart on the contractors team, but he sent this fantastic one line note to me an hour later (with a CC back to original director):
I am going out now so please follow-up with [Team Lead A] & [Team Lead B].
Again, I have to ask, who is running this project? Me? Because if it is, where's the fucking money? And why can't I fire the 3/4 of the dev team who consistently fuck up? The ones who contribute little or nothing to the project? The ones who actually cause more issues than they resolve? The ones who wrapped perfectly good checked exceptions inside of useless unchecked exception because it was "easier" (for you non-programmers, it's like wrapping a hammer in a feather pillow - completely ruins the tool for what it's supposed to be used for)? Why can't I impose process? Why can't I demand the "A game" from the other teams?
Why am I the one getting yelled at for being unprofessional?
Why am I the one being criticized for not taking ownership?
More rope?
Anyone who's been reading this for the last couple of months knows that I work with, and let's be polite, Gumbies. Here are my coworkers in action:
Yesterday, I went to lunch with some of my fellow team members. I got to talking with one of the few competent devs on this project, a lovely young lady, and Spurs fan (she's young, she'll learn eventually). One of the Gumbies is her team lead. We got to chatting about the deployment date for our application. Our current deploy date is the 19th. I don't have high hopes of it. But I am doing my best to meet the date, because that's what I do. It's OK to doubt the date. If you're an adult, you are probably capable of being trained. And we've all been trained on this project to doubt the date using simple repetition techniques. But we're also responsible for being ready for the date, just in case.
Not so for Gumby Team Lead. She told her team members that the 19th probably wasn't going to be the date. Oh, sure, officially, it is, but it probably won't be, she said. She took pains to stress that. So now, her team doesn't think they have to meet the deadline. This dev I went to lunch with was confident that the 19th wasn't the date. It's not surprising, really, since Gumby's team has the largest number of chronic deadline flaunters on the project. Most of her devs couldn't meet a deadline if you threatened them with a chainsaw. And now we know why: deadlines mean nothing to them, because their Gumby Team Lead has made it clear that there are absolutely no consequences for missing a deadline, whether it's a daily schedule or a project timeline. The 19th? Don't worry about it, it doesn't really matter.
We need to send her for brain surgery. And I really want to be the anesthetician.
Vito and I were laughing so hard yesterday, I started having a coughing fit.
Why?
More rope.
See, the development team "leadership" decided that Vito and I, who used to be called build managers, would now be more like build lackeys. They had a meeting on Monday, with no build managers present, where they decided to not have a build schedule. They'd just ask for builds whenever, and, well, Vito and I would simply have to do them. After my less than useful meeting last Friday, and Vito's experiences with attempting to enforce any kind of process, we decided, what the hell, let's just do it and see how long it takes to seriously screw something up.
You know how in those pirate movies there's always that one scene where the captain is fighting the wheel in a storm? And then something happens and he lets go of the wheel? And the wheel just starts spinning out of control and the ship sinks? Well, we're not sunk yet, but that wheel could power lower Manhattan for a month.
We did 10 builds and/or deployments between 11 AM and 7 PM. Every build got worse, because no one knew what was supposed to be fixed by what time. We had complaints from the developers because every time we do a build, they have to update their copy of the code from our code management tool. Each update can take up to 20 minutes, depending on how busy the server is. Considering the build load, I expect it was pretty busy. Updates block all other activities. So that's, let's see, 20 minutes times 10 . . . heck let's just call it three hours per developer of lost productivity. Yesterday. Not over a whole week. In one day.
More rope.
And the builds? Are they even being tested? How can they be? A deployment takes time. The servers have to come back on line. In one instance, Vito finished the automated test scripts and was asked, before he was able to send out the results, to do another build. There's no way that was actually tested to confirm the supposed fixes were working.
More rope.
Almost every build we did had to be done twice because some developer had some extra code to check in. Why? How can we blame them now? There's no schedule for them to even try to adhere to, even if, for the most part, they didn't adhere to it when we had one.
More.
Rope.
We got a suggestion from one of the team leads to put up a sign with the build schedule, because all the e-mails we were sending out were confusing (we announce the build, the deployment and the test resutls - that's at least 30 messages every person on the development team got from us in 8 hours). First of all, what schedule? Second of all, half the devs are on the other side of the building, and won't see the sign. Our counter proposal was for me to hold the sign above my head and walk around like one of those girls at a boxing match announcing the round number. I even offered to wear a bikini. The other counter proposal was to have one of those scrolling LED signs to announce the build schedule like a stock ticker. I'm actually working on that now, to put it on our build SharePoint site. It's required me to get Visual Studio and learn how to make Web Parts. It's a waste of time, because no one is ever going to bother looking at it. But, let's remember, I'm a build lackey now: they ask, I do, that's my role.
More.
The crowning achievement in the world of absurdity was this: Vito, very slowly, very carefully, very clearly, explained that each build was getting worse because there was no coordination, and no one could tell when anything was going to happen, or what fixes were supposed to go in which build. Case in point, one of the introduced failures was because of some updates that went in without the knowledge of any of the team leads because the team leads spent all day asking for builds, rather than LEADING their TEAMS. The team lead's response? "Why is it like this?" My jaw very nearly dropped right off my face. Vito just explained it to you. Do you want pictures? A pop-up book? 101 Ways to Fuck a Project in the Ass? Perhaps a copy of Listening Skills for Fucking Retards?
Rope.
Vito and I were literally falling out of our chairs laughing about this.
The best part? I worked 15 hours yesterday. Fat load of OT. And we accomplished exactly nothing. And today looks like it's going to be another side-splitter.
I really don't like being at the tail end of a development stream. Before moving away from development, the code I was writing was at the end of the process. This means that I was reliant on everyone up at the beginning of the process to do their jobs and get things done on time. They did not. This meant, strangely, that I was the one punished, with shorter delivery times, and random coding changes when those early process jerks made a code fix. Never mind that dozens of other developers were relying on that code, just go ahead and change it.
It's like designing a whole car, and then, at the last minute, the jerks working under the hood decide to change the engine and now everyone from the people making the motor mounts to the guys working on the transmission have to completely revamp their code, working frantically through the night to adapt to the whim of some ass hat decision. Meanwhile, the engine fuckoffs are over at the side of the garage drinking tea and complaining that no one supports them.
Sadly, if anything, it's worse now. I'm not required to do any development, but the same idiots are making my life miserable by constantly breaking the build, or asking for another build, or checking in code that, well, sort of works on this deployment, but not at all on another deployment. And who has to stay up until midnight? Not the coders, oh no, those fucking pukes get to go home at 6 while I'm working the night away. And every time we (I'm not the only builder, but we're outnumbered 25 to 1) try to hold a line and say, guess what, you fucked it up, I'm not sticking around to wipe your ass, suddenly, it's a national emergency, and the development leads get involved and we're backed into a corner.
It's all about to come crashing down, though.
My colleague has finally started showing signs, after months of this, that he's fed up. So, we're planning a major twin-pronged attack: we're both going to be out one day. Just some random day. We're both sick. Our cell phones are mysteriously off. No e-mail contact. And after that chaos, we'll come back. And the next time someone complains about the build schedule, or breaks the build, or tries to whine and weasel their way around the process, we'll say, remember March 10th? Remember that day when there were no builds, and no deployments and no one could get anything done or tested or fixed?
Want another one?
Then sit the fuck down and do your fucking job. And I'll do mine.
It's been almost a week since my last post. And what a week it's been.
I asked a co-worker to update some code based on sample code I gave her. She sat on the code for 5 hours, then e-mailed me after I left work to ask if she really needed to update the code because the example had the same numbers in it that her code already had, and isn't this a waste of time? Of course, had she been listening when I told her what the sample code was for, she wouldn't have had such a moronic question for me. And now, the code isn't updated.
I was told to use a connection I was already using because the connection was failing if I was using a connection I wasn't using. I know who was using the connection in question: everyone on the project. And I know why: when I asked what connection I should put in the global setup document, that was the one they gave me. If you're going to complain that everyone is using something no one should be using, then maybe you shouldn't tell me to tell everyone to use it.
I got an e-mail from my team lead which said, pretty much verbatim, I'm not getting e-mails when I try to send e-mails from the application. Well, there are four ways to send e-mails. Which of the four ways did you try? Or did you try all of them? I happen to know for a fact that, provided I am handed a proper contract from the UI (where that first co-worker decided to argue with me about the update rather than listening to me in the first place), e-mail works. Faxes work. Everything I've been told to create works fine. Or at least it does now that I've taken out all the crazy ass test code that some other developer put into my code for some unknown reason. And it worked before I left on Friday. That was really my task for the day, and I made sure I completed it before I left. The fact that the UI isn't passing the right contract, or that some other developer put back in the dumb ass test code isn't my problem. Further, if you're going to complain that something for which there are four ways of doing it doesn't work, you'd better tell me which way you attempted to accomplish said something. If I go to my doctor and say, "Hey, doc, I can't pick up my keys," I suspect his first question is going to be, "Which hand did you try it with?" If I then come back and say, "Well, no, I was trying with my teeth," I would fully expect my doctor to say, "Look, moron, if you're going to complain to me that you can't do something, you'd better tell me what you tried to do it with."
But best of all is this: We were supposed to be code complete last Thursday. We had a plan for Wednesday night to lock out the development on all but one stream, and to merge everyone's work together. We locked the stream, I went home. I get in in the morning and discover that the stream had been unlocked for a further 3 merges, but no one is owning up as to who requested that. I'm now trying to do damage control and get things back on track. I had been told that there is exactly one person who can approve code that breaks previously working functionality, so when I need to ask him, I just walk in on a meeting and ask. I don't have time to waste, we have a schedule. Well, later, I get a reprimand from Team Lead X. I shouldn't just barge into meetings. I should wait, or ask someone else. Well, that's not what I was told. I was told this is urgent, priority 1, and I can't ask anyone else. If you're telling me now that I can ask someone else, fine, but don't get all bent out of shape because I am doing what I was told.
Turns out the person who asked for the streams to be reopened was Team Lead X. I found this out on Friday. Why? Because two devs didn't get their changes into the locked stream in time. Guess what? Too bad. It's locked. Move your changes to another unlocked stream by yourself. It can be done, unless the dev in question is so big a moron that he makes Elmo look like Stephen Hawking. In retrospect, the dev in question is just such a moron. Still. It was locked. Had I known that before the reprimand meeting, trust me, that meeting would have gone very differently. Along the lines of, "I really don't think I'm going to take protocol lessons from someone who broke the delivery protocol because he can't manage his developers."
I've got the last laugh though. They pushed me to take on more responsibilities. Presumably they thought I'd just roll over and do as they asked. So, I joined the build team. The team is small. It's Christmas week. The office is pretty empty. I'm the only one who can do builds today. And I have my own schedule to keep. And I'm keeping it. Complaints? Like Lando complaining to Darth Vader, all I have to say is . . .
I am altering the deal. Pray I don't alter it any further.
So, I can't get any work done because our lovely source control system which can't handle simple merges without undoing the efforts of the entire team doesn't have enough licenses to go around.
Yes, I can see where open source software is a real hassle, what with CVS not requiring any licenses and all. I'm sure users of Subversion must be constantly tormented by the utter lack of work stoppages due to license shortages. Oh yes, this is a fantastic feature, and I am certain the company will be glad they invested a lot of money, but not quite enough, in ClearCase when it comes time for us to deliver this project and it's not done because half the team had to wait for the other half to make updates and log out.
OK, so, once again, we have a change to our database. This is, of course, just the latest update in a long string of updates to our database schema. I am certain that someone somewhere knows what sort of insanity it is to be futzing around with your database layer a mere 4 weeks before code is supposed to be delivered, but apparently that person doesn't work for our current project.
So, it fell to me to make the update. Why? No idea. Guess they decided I didn't have enough else to do. Or maybe they thought that since the last 3 updates were done primarily by someone else, I'd be a good candidate.
I worked through the updates, worked through the errors that resulted, tried to have a low impact on the process, but, in the end, I screwed it up. Some of the updates didn't work, by the simple issue of the service we have to build the files using a config file that wasn't 100% up to snuff. I did my best, I failed. I took the output, compiled against it, ran what test cases I could run, and made sure there weren't any errors to speak of.
Unfortunately, one of the areas of interest turned out, in the end, to not work. So, in addition to cleaning puke off the floors, couch and bathroom fixtures, I spent part of yesterday fixing my mistake. Or so I thought.
It turns out, once I fixed what I presumed was the mistake, and checked that back in, that there isn't an available test for the code in question, and never was. Our offshore team was using a test that they hadn't checked in to source control yet. So, I asked them to run their test against the build stream I had checked my changes in to, because said build stream, call it stream A, was the only one with the fix.
This morning, I get the following e-mail: "We couldn't run the test against stream B."
I immediately returned with the following: "Run it in stream A. Stream A is where the changes are. If you can't run it in stream A, ask someone who has access to stream A to run it for you. STREAM A."
I think they may have gotten the message, because it's gone a little quiet on their end while they update stream A to catch my changes. What's annoying is that I somehow decided that this was more important to fix than paying attention to my daughter. I'm disgusted with myself over this, and I'm not sure what to do about it.
I've discovered a rather good site where I can vent my frustrations in a creative manner. Here's my collection so far.
This one is pretty self-explanatory. Actually, they all are. The issue here is the continued penchant many of my so-called co-workers have of supplying me with bug fixes that not only haven't been tested to ensure, you know, that the bug is fixed, but generally tend to cause more bugs. I've taken to running my test cases from the highest possible level, but even then I can't be sure, because I can never get a straight answer from the people developing the UI about what they are, in fact, handing to the server. I asked, hey, are you setting the user's name in the "userName" field? They say yes, I test that way, then post the fix. When it gets retested, it turns out the name was actually being handed to me in the "nameOfUser" field, which seems to me to be a duplicate, but, whatever. Thus, the bug still exists, albeit in a slightly different form.
The better part was when one of my good co-workers and I spent two weeks generating a large data set which was then summarily deleted because of an update to the database layout (or "schema" as it's called by DBAs). I can't wait for an update like that to production code that wipes out the system of record. SarBox violation, anyone?
These two sort of go together. Our merge problems are so massive, it's gone beyond a joke into the realm of, guess what, we're never going to finish this if you don't change your ways. Hence, the two posters you see here.
Seriously, it's horrid. Automerge often blows up code that was working before, because it has no concept of dependency checking. I've lost almost as many fixes as I've saved. Today, I'm working with the team to deal with automerge issues that arose from code that was auto-generated by our Web Service code generator, which was foisted upon us because we have a service contract, not because it actually meets our needs. But I couldn't find an appropriate picture for that.
And this bad boy, well. I can tell you that we do actually have version history maps of certain files that look, if anything, worse than this. Because our delivery plan is miserable. That's not just my opinion, but the opinion of pretty much every competent coder on the team. It's a waste of time.
We were told today that, to avoid automerge and delivery problems, we should wait after we make a fix in one stream for those changes to be delivered to the other streams. Because, you know, if there's something you really want to do to make sure you meet your deadlines, it's forcing your developers to wait on a bad process.
Last but by far not least is my personal favorite. We have, for a long time, had major issues with people checking in code to our build server that not only doesn't work properly, it doesn't even compile. This means phone calls late at night to our offshore team, and random fixes that, given the above issues, are really not advisable. And even if they were advisable, probably wouldn't survive the next code merge.
So, I leave you with this.
We have a very odd (by which I mean cumbersome and ineffective) system for managing code releases. I understand the theory, of course. You have a dev stream, and a test stream, and, in our case, a system stream. Ideally, you do the new development in one stream, bug fixes in another, and the system stream is relatively stable. Code gets merged back once in a while, and you leave the really major bugs to be fixed by hand in both the dev and test streams. We just cut our system stream yesterday.
But that's not how it's working around here. We're doing a time-tested kind of development. Not RAD. Not cascading. Not even straight build-test-deploy. No, we're doing development in a style as old as time, and one which many development teams end up in no matter how they started out: holy-shit-this-is-due-in-5-weeks-seat-of-the-pants development.
Now, if you're IBM, and you need to justify your massive bills to the customer for making their lives harder and not really doing anything, you ensure that you waste as much of everybody's time as possible. This means that we're building in dev, bug fixing in test, and, well, I don't know what we're doing in system. But I do know this. We're merging our code from all three streams every day.
That's just plain stupid. Or, as I like to call it, IBM SOP.
If you're merging your code daily, why have three streams? There's no point, other than to make the lives of the devs more difficult, and to rack up a bunch of overtime and say, look at all the good work we're doing, when, in fact, you're fucking us all as hard as you can and expecting us to thank you for it. Oh sure, it allows bug tracking. But when your merges randomly remove bug fixes, how, exactly, are we tracking them again?
If we're merging back in every day, why not have one stream? We're all reasonably intelligent people. Well, most of us. Well, OK, the devs who are actually competent enough to make a good bug fix are able to both fix their bug, and do new development in the same stream. It's possible, is my point. This three stream thing was deleting bug fixes and breaking builds in dev when it was only two streams, because ClearCase's automerge is a joke. We don't have time for this crap any more. Kill the other two streams, and make daily cuts from the dev stream to go to the three environments.
Which, by the way, would ease the other issue we have, which is that all three environments have slightly different set ups. One DB schema has restrictions that another doesn't. We're running some things on Windows, and some on Linux, and some on AIX, and no one has bothered to ensure that the environments are in the least bit similar. Which is why my latest bug can't be replicated anywhere except on the test servers, which, since we have this stupid three stream set up, means I can really only check to see if my random guess of a bug fix works once a day. Oh, and that new cut? The one now deployed to system? Yeah, it doesn't even load at all, because the DB isn't even using the same schema as test or dev. Nice. But if it were all cut from one stream, well, there'd be no excuse. No temporary fix for this environment or that environment, we'd have to have three environments that are basically the same, which, well, is sort of something that you want to have anyway. I mean, if you want to get this project done in mid-December. Makes no never mind to me either way.
But no. We have to cling on to the way that IBM fucked this project before, because we're so fucking entrenched now we can't see that just over the hill is a glorious new dawn of easy development without stupid baggage if only we'd all stop for two days and think about what we're doing, and just fucking fix it. Starting with firing the last IBM guys: the ones running the ClearCase servers and forcing the team to do this dumb ass three stream development.
Programmer nerds, read this. Especially if you're working with typical offshore resources. What's got me set off today?
DeliverPricingInfo pricingInfo = null;
// 3 lines of irrelevant logging code
//Save the Pricing delivered
pricingInfo.save(pricingInfoContract);
What kind of rookie writes code like this and thinks it will work? And who allows it to be delivered as working code? This can result in only one outcome, and would be caught easily during even the most rudimentary testing. Like, say, a simple peer code review. Clearly, no one is awake in our offshore development center. Code like this is going out to ride your bike, seeing that it has no wheels on it, and thinking, "Yup, this will work." This isn't square wheels, or triangles, or a broken chain, or handlebars turned the wrong way, or some funky recumbent design that no one's ever tried before, or an uncomfortable seat designed for a baboon's ass, this is no fucking wheels. And thinking it will work.
You're doing it completely wrong.
