Word

And because that last post had so many words, here’s a shorter one about, well, words.

A few key phrases to know if you ever attend a meeting in the GooglePlex:

Orthogonal
Engineers are always talking about things being orthogonal to each other. The first time I heard the term, I thought it meant something like “11-sided.” It doesn’t. I’ve read the definition many times. I still don’t really get it, which didn’t stop me from casually dropping it into conversations with engineers. “Oh, yeah, that press release is totally orthogonal to the ads we’re running on Yahoo.”

Cruft
Cruft is bad. Like the stuff that grows under ungroomed toenails. Like barnacles on a speed boat. It usually refers to old code or dead links on a web page, but it can be applied to any unwanted material that accumulates anywhere. The men’s locker room in the Googleplex was filled with cruft, much of it unwashed and hockey-related.

Canonical
When engineers don’t want to say “the customary” or “the usual” way of doing something, they talk about the canonical example, which is pretty much the same thing, but conveys a more exalted sense of correctness to the practice in question. The canonical way to avoid spending an hour in traffic is to come to work at noon.

Non-trivial
It means impossible. Since no engineer is going to admit something is impossible, they use this word instead. When an engineer says something is “non-trivial,” it’s the equivalent of an airline pilot calmly telling you that you might encounter “just a bit of turbulence” as he flies you into a cat 5 hurricane.

So…
This all-purpose word is not a word at all. It’s the sound of an engineer clearing his or her throat before beginning to speak. The first week I worked at Google, it seemed like some sort of linguistic virus had infected all the technical staff. Every sentence in every conversation began, “So…” So… I eventually got used to it.

Six years done gone

I just realized that yesterday would have marked my 6th anniversary at Google.

Perhaps because the company was still so new, no one made a big deal out of anniversary dates in the early years, with the exception of the first one. As Ron explained, twelve months was your cliff vesting date, which meant that you were now the official owner of the first quarter of your stock options. After the year mark, your stock vested on a monthly basis.

I remember one engineer wearing an actual vest on his vesting day, but generally, it wasn’t a big topic of discussion. No one talked about how many shares they had been granted or what they would buy once the company went public. That may have been because it quickly became apparent the company wasn’t going public any time soon.

When I joined in 1999, employees generally assumed that Google would be public within a year. Googlers weren’t obsessed with the idea of an IPO, but it was the height of the dotcom era, and that’s what startups did after they’d been around for six months or so. Larry and Sergey didn’t seem in any particular hurry though, and they laid out a long checklist of items that would need to be completed before we could even consider such a thing. Hiring a CEO and a CFO were the first two to-do items, which I believe was largely driven by the VCs on Google’s board.

It became a running joke at TGIF after a while. Bart from Ad Ops or Paul or Schwim from engineering would stand up during the Q & A session and ask where we were in the CEO search process. Sergey would smile and say we were “making progress.” Sometimes he’d make a dismissive remark about a candidate they had interviewed because he lacked adequate technology chops or indicate that another seemed “interesting” because he had taken them helicopter skiing.

In early 2001, Eric Schmidt was brought on board as CEO to great acclaim, not necessarily because people could see from the start that he was a perfect fit, but because the long wait was finally over and we could check one more thing off the pre-IPO countdown.

Then began the even longer hunt for a CFO and Eric quickly got into the spirit of things by letting everyone know we were “making progress” whenever anyone asked about the search. He also introduced a kind of continuously phase-shifting chronological ambiguity by responding to all questions about the IPO with the stock answer, “six to nine months.” As he put it, “Whatever today is, the answer will always be six to nine months from that date.”

That went on for years, until we got close enough to the IPO filing that we entered something like radio silence, with the execs not commenting extensively on what was going on, though everyone had a pretty good idea.

During that period right before the IPO, there was a sudden flurry of activity around employee retention and a service awards program was one of the ideas thrown out as a way of recognizing Googlers and making them feel more loyalty to the company. I don’t know if the program was ever implemented, however.

So, six years later, I don’t have a piece of Tiffany crystal from Google like the one I received after six years at the Merc, but I do have fully illuminated memories of some dazzling moments etched in my mind. Some were crisp, clear and brilliant, others more a pattern of light and shadow that could be interpreted many ways. The latter will be hard to paint with a few paragraphs in a blog post. I’ll do my best to capture them accurately, knowing that others likely have different views, which they’re welcome to post here if they wish to set the record straight.

On that note, it’s been interesting to read opinions about whether I’m an apologist for Google or a disgruntled ex-employee with an axe to grind. I don’t consider myself either. I’m a guy who happened to be standing on the platform with an open-ended ticket when good fortune steamed into the station. I can’t even claim I knew where I was headed when I climbed on board. I wasn’t driving the train; not even stoking the boiler. I was more like the guy yelling out the stops as they approached. Hardly an essential role, since Google would have arrived anyway, but one that let me look ahead and tell the passengers what wonders they could expect.

Google wasn’t paradise and I’m not blind to its faults. As Ron has already noted, some things did go wrong. I’ve got a few anecdotes on that score myself. But looking back down the track, I have remarkably few regrets. Google started as a small group of people trying to solve big problems, and the more I got to know them, the more I respected their integrity and their ambitions. I’ll always feel lucky for having worked with them.

Racing to the finish

Ron sez... oh wait, don't need that any more.

OK, time to wrap up this little soap opera.

The problem turned out to be something called a race condition, which is one of the most pernicious and difficult kinds of bugs to find. (Those of you who are technically savvy can skip to the end.)

Most modern server code is multi-threaded, which means that it does more than one computation at once. This is important because computers do more than just compute. They also store and retrieve information from hard disks, which are much, much slower than the computers. Every time the computer has to access the disk things come to a screeching halt. To give you some idea, most modern computers run at clock speed measured in gigahertz, or billions of cycles per second. The fastest hard disks have seek times (that is, the time it takes the drive to move the read/write head into the proper position) of several milliseconds. So a computer can perform tens of millions of computations in the time it takes a hard disk just to get into position to read or write data.

In order to keep things from bogging down, when one computation has to access the disk, it suspends itself, and another computation takes over. This way, one computer sort of "pretends" that it is really multiple computers all running at the same time, even though in reality what is happening is that one computer is just time-slicing lots of simultaneous computations.

The ad server, the machine that actually served up ads in response to search terms, ran multi-threaded code written in C++, which is more or less the industry standard nowadays for high-performance applications. C++ is byzantine, one of the most complex programming languages ever invented. I've been studying C++ off and on for ten years and I'm still far from being an expert. Its designers didn't really set out to make it that complicated, it just sort of accreted more and more cruft over the years until it turned into this hulking behemoth.

C++ has a lot of features, but one feature that it lacks that Lisp and Java have is automatic memory management. Lisp and Java (and most other modern programming langauges) use a technique called garbage collection to automatically figure out when a piece of memory is no longer being used and put it back in the pool of available memory. In C++ you have to do this manually.

Memory management in multi-threaded applications is one of the biggest challenges C++ programmers face. It's a nightmare. All kinds of techniques and protocols have been developed to help make the task easier, but none of them work very well. At the very least they all require a certain discipline on the part of the programmer that is very difficult to maintain. And for complex pieces of code that are being worked on by more than one person it is very, very hard to get it right.

What happened, it turned out, was this: the ad server kept a count of all the ads that it served, which it periodically wrote out to the database. (For those of you wondering what database we were using, it was MySQL, which leads to another story, but that will have to wait for another post.) It also had a feature where, if it was shut down for any reason, it would write out the final served ads count before it actually quit. The ad counts were stored in a block of memory that was stack allocated by one thread. The final ad counts were written out by code running in a different thread. So when the ad server was shut down, the first thread would exit and free up the memory holding the ad counts, which would then be reused by some other process, which would write essentially random data there. In the meantime, the thread writing out the final ad counts would still be reading that memory. This is why it's called a race condition, because the two threads were racing each other, with the ad-count-writer trying to finish before the main thread freed up the memory it was using to get those counts. And because the ad-count-writer was writing those counts to a database, which is to say, to disk, it always lost the race.

Now, here is the supreme irony: remember the meeting with Larry where he wanted to make a change to the billing model that I said would be hard and everyone else in the room thought would be easy? The bug was introduced when the ad server code was changed to accommodate that new billing model. On top of that, this kind of bug is actually impossible to introduce except in a language with manual memory management like C++. In a language with automatic memory management like Java or Lisp the system automatically notices that the memory is still in use and prevent it from being reused until all threads were actually done with it.

By the time this bug was found and fixed (by Ed) I was a mental wreck, and well on my way to becoming a physical wreck as well. My relationship with my wife was beginning to strain. My manager and I were barely on speaking terms. And I was getting a crick in my neck from the chip I was carrying around on my shoulder from feeling that I had been vindicated in my assessment of the potential difficulties of changing the billing model.

So I went to my manager and offered to resign from the ads group. To my utter astonishment, she did not accept.

Code poetry

Your many suggestions on how to make this blog more readable are helping. I fought my Luddite tendencies and played around with the template enough to get the names at the top and the ads on the side. Now, if suddenly everything goes kerflooey, you'll know why they didn't let me write code at Google. Though there was one time ...

The receptionists at Google always seemed overqualified for the tasks they were given. They smiled and pointed guests to the cooler full of free Naked Juice, explained how the massage chairs in the lobby worked, dialed the extension of the person being called upon and then consoled the visitor for half an hour or so until the Googler in question showed up. One result was that when given the opportunity to express themselves in more intellectually stimulating ways, they did so.

I think it was Deb who started emailing notices of lost and found objects in verse. One evening, this message arrived in inboxes across the network:

It was all alone, this sweet little phone,
And it went by the name of Verizon.
Silver and light, respond to its plight!
Please retrieve it at Bayshore Reception.

In a whiff and flutter, the scarf was a-hover,
And lost its way on one googly afternoon.
It's chenille, true -- and if this sounds like you,
Come retrieve it in the Bayshore Reception room.

Wei Hwa, an engineer and four-time winner of the World Puzzle Championship, responded in kind:

On a day such as this, one so merely mundane,
came a double epistle with a common refrain.
Assorted lost items with no one to claim,
inspiring lines that'd put Byron to shame.

Ranging from sane to the slightly absurd,
A motley of rhymes -- nay, a true smorgasbord;
The vocals! The echoes! The choice of a word --
but I think that the truth is: you're just awfully bored.

By the time I saw Wei-Hwa’s note it was after 10PM. As usual, I was logged into Google’s VPN from home, working on some project or other that was launching in the morning. I couldn’t resist sending this message:

TIME = -1

GET FILE(SYSIN) LIST(NUM):

N=0

DO CNT=1 TO NUM:

GET FILE(SYSIN) LIST(ROW,COL,TIME(ROW,COL));

 IF COL>N THEN

 N=COL;

END;

DO ROW=1 TO N;

 DO COL=ROW TO N;

 IF TIME(ROW,COL)>=0 THEN

 ES(COL)=MAX(ES(COL),ES(ROW)+TIME(ROW,COL));

END;

 END;

PUT FILE(SYSPRINT) SKIP EDIT('CRITICAL PATH ACTIVITIES')(A);



If engineers get to write poetry, English majors get to write code.
 Btw, our own esteemed Peter J. Norvig gave me an A- on this assignment when he was a grader for my CS 50 class. Oh, what might have been...

Within seconds a dozen Googlers, including Eric Schmidt, emailed me to critique my code, ask what dust heap I had found it in and apprise me that no living coder on the planet still used this programming language. My favorite response came from Meng, an engineer, who simply noted:

Good tight loops,

Assignments look fine.

Logic flows well,

Correct every line.



Not too bad,

For English major.

Sadly, my friend,

Bart codes better.


The night in question was not extraordinary by Google standards. You could always count on reaching the people you needed when you needed them. And when you did reach them, they usually had pretty interesting things to say.

Well, of course for interviews, you dress up...

Doug again...

Google's engineers each possessed unique qualities that made one appreciate the diversity of forms a logical mind can assume. Ray's qualities were perhaps the outliers on the scatter plot of that diversity. Infamous for the fluorescent pink nylon shorts he wore to work, Ray recently forwarded his own recollection of interviewing Chad, another candidate who landed an engineering position at Google:

I had told the HR people that maybe they shouldn't schedule interviews on Halloween. But they gave me two in 1999, and, yah, I was a smurf. Chad was one candidate that day, and the other was someone we didn't end up making an offer to, I think.

The problem with dressing up as a smurf is that the paint sort of comes off your hands, and so every time you go to take a leak, your smurf turns a little bluer.

I remember Sergey wearing a cow suit one year. Dunno which! I have some vague notion that it might've been Salar's cow suit at the start of the day, and then Sergey took a shine to it. But perhaps not.

Another Googler did, in fact, tell me once about answering interview questions as Sergey, attired in a full-size cow suit, absentmindedly stroked his rubber udder. In retrospect, a roller hockey getup seems fairly formal by Google standards.

Google's tradition of outrageous Halloween costumes was nourished by Heather, the world's most politically incorrect HR manager. A former flight attendant and avid surfer, Heather's costumes typically took on disaster or topical themes involving plane wreckage, sharks, altar boys and lots of blood. I think she may have inspired Bart, who at six foot six, made an imposing figure dressed in a leather S&M mask and baseball uniform as he declared himself to be Barry Bondage.

I went as Deputy Director Skinner from the X-Files one year, since I already had the hairline. Someone saw a photo of me talking to a sumo wrestler and posted a link to it on Slashdot, asserting that the presence of an FBI agent at Google's Halloween party proved Google was in cahoots with the feds to spy on American citizens. I took it as a great compliment to my costume's verisimilitude.

All of this is by way of letting you know that we are now joined on Xooglers by the distinguished Heather Cairns from Google HR, one of the company's earliest employees and the first person a Noogler met on the day he or she started. Heather's droll commentary as she explained Google benefits and the corporate culture sometimes caught the wide-eyed neophyte by surprise. It sometimes amazes me that so many Nooglers came back for day two. I look forward to once again enjoying her fresh perspective and sardonic wit now that she's here on Xooglers. Heather?

TGIF II

Doug continues:

The week after my introduction at TGIF, I brought my video camera to capture for my parents back in Jacksonville the flavor of a Silicon Valley startup. I caught most of the Friday meeting (Larry didn’t want the board slides filmed), which went something like this:

Joking and smiling Googlers assemble in the hall outside Larry and Sergey’s joint office as Yoshka ambles around in search of a Milkbone and Susan tries to keep him from sniffing her newborn, who she carries in a car seat.

Charlie enters in his white chef’s coat and apron, carrying a tray of chicken wings, which he sets down next to Craig (employee number one), who is opening and distributing bottles of beer. Three engineers jam on their computers, sitting side-by-side-by-side in a single glass-walled office in the shadow of a large K’Nex tower.

Sony monitors stacked in their shipping cases form a backdrop as Ray walks his skates back to his office and Larry, dressed in black slacks and a long-sleeved dark blue shirt buttoned to the collar, steps to the front of the crowd.

“Welcome to our new and exciting TGIF…,” he begins, before introducing Sanjay, who has a masters and PhD from MIT and is tall enough that Sergey jokes we’ve begun recruiting for height. Keith in accounting is introduced to polite applause.

Larry reaches down to switch on a projector sitting on a shaky white plastic table and begins walking through the board slides. When the camera goes back on, Sergey is calling out the names of those celebrating birthdays this week, including Larry. The honorees move to the three cakes (coconut crème, chocolate and strawberry) and as the first slice is served, there’s suddenly Silly String everywhere. Sergey is handing out cans which are soon emptied, with hair, shoulders and plates all dripping with the stuff.

Sergey calls for quiet and begins talking about a dilemma he has faced, “What to get Larry for his birthday.”

He continues, “I always noticed he'd pass my office with a gleaming look in his eye. I took it as a sign of affection, until I realized he wasn't looking at me, he was looking at my throne. As CEO, Larry deserves his own throne.”

With that, a large box containing a purple plastic gamer’s chair with built in speakers is pulled out from behind a grey fabric-covered wall. It’s identical to one that Sergey has been using.

Larry thanks Sergey, who wishes him a happy birthday and the video ends.

Well, it may not get a standing ovation at Sundance, but it does do justice to the tenor of the company in late 1999: the informal atmosphere, the shared information about important concerns and the fun of just kicking back with good food, beer and friends to celebrate special occasions. That went a long way for a long time at Google until rapid growth and raised expectations inevitably began creating stress points.

I'm sure that in ten years, the Google of birthday cakes and board slides and purple chairs is the Google I will choose to remember most clearly. I hope it will always remain a part of the culture in some way, though I fear that the hard-charging uber-achievers now flooding the company in such great numbers may not make it quite as high a priority.

Perhaps they would be right not to do so. Frivolity is hardly efficient and comraderie can't be quantified in a way that will help win the upcoming wars with Microsoft and, well, everyone. Still, I urge those old-timers haunting the halls of the 'plex to share their experiences with those more recently arrived, and perhaps to dig out the old Nerf guns, the Zappies, the Soul Caliber and the Big Wheels from wherever they've been closeted and give them another go.

And maybe someone could ask Larry whatever happened to his purple chair.

I say to myself, 'My God, what have I done?'

Doug again:

Ron’s posts are stirring a lot of memories for me, as well as interest in which blowup he’s leading up to. There were a few that I was only vaguely aware of from the marketing side. Not that they were hidden from me. To the contrary, I probably was in the meeting he described, my eyes glazed over as the talk switched to full geek-speak, filled with failed pushes, gws crashes, mis-assigned port numbers or slow server synch-ups. Just writing that makes me want to take a nap.

Questions about the level of communication within Google get asked a lot. Because the company maintains a great wall of secrecy between itself and the outside world, it’s natural to assume that Googlers themselves don’t have a clue what’s going on in their own company. That may in fact be the case now (I’ve been gone eight months), but in the early days, everything was out in the open, from how much cash was left in the bank to exactly how many times the VCs had requested we hire a CEO in the last board meeting.

This openness continued for a much longer time than one could reasonably expect, primarily because people kept everything confidential, despite Ben Franklin’s dire prediction.

Most of this information was shared at TGIF, the weekly 4:30 gathering that Larry and Sergey led as Googlers sat on the floor or bounced gently on brightly-colored exercise balls (the round ones were far more comfortable than the peanut-shaped ones). The format remained the same for quite a while, with new Googlers introduced first, then a wrap-up of departmental news, a Q&A session and finally a celebration of those Googlers having birthdays, replete with cakes, beer and Charlie’s choice cuisine.

Shortly after I joined, I had to introduce myself to this assemblage of 60 or so people using the approved template. I still have my notes:

Your current title:

• Online Brand Manager


Your current responsibilities:

• Building our brand online and driving traffic to our website.
• Keeping Aydin company.
• Taste-testing everything in the cafeteria.


Where you were employed prior to Google:

• San Jose Mercury News as Online Brand Group Manager.


Your education background:

• Bachelor's degree in English and American Lit. from Brown University


What you do in your free time:

• Spend time with my wife and kids.
Anything else fun/interesting/fascinating that Googlers should know about
you:


• I draw very morbid looking skeletal doodles in meetings, but it's not really reflective of the state of my mental health or how closely I'm paying attention to what people are saying.

As the number of new employees jumped from one or two every couple of weeks to several every week, to eventually, a seemingly unrelenting tide of talent, this ritual was cut back to a displaying a brief snippet and badge photo flashed up four at a time on Keynote slides. I produced TGIF during this period and will likely come back to the joys of that challenge at a later time.

I was nervous as I ran through my paces and couldn't help noticing that I was from a different decade than most of those standing around me. They smiled and applauded and generally made me feel at home, but I wondered, not for the first time, what exactly I had gotten myself into. Like Ron, I had made a financial commitment that stretched my limited means by joining this company and my investment rested in large part on the grey matter behind these happy, shiny faces. I prayed that they had a better idea of what they were doing than I did.

The billing disaster

The AdWords launch went fairly smoothly, and I spent most of the next two weeks just monitoring the system, fixing miscellaneous bugs, and answering emails from users. (Yes, I was front-line AdWords support for the first month or so.)

The billing system that I had written ran as a cron job (for you non-programmers, that means that it ran automatically on a set schedule) and the output scrolled by in a window on my screen. Everything was working so well I didn't really pay much attention to it any more, until out of the corner of my eye I noticed that something didn't look quite right.

I pulled up the biller window and saw that a whole bunch of credit card charges were being declined one after another. The reason was immediately obvious: the amounts being charged were outrageous, tens of thousands, hundreds of thousands, millions of dollars. Basically random numbers, most of which no doubt exceeded people's credit limits by orders of magnitude.

But a few didn't. Some charges, for hundreds or thousands of dollars, were getting through. Either way it was bad. For the charges that weren't getting through the biller was automatically shutting down the accounts, suspending all their ads, and sending out nasty emails telling people that their credit cards had been rejected.

I got a sick feeling in the pit of my stomach, killed the biller, and started trying to figure out what the fsck was going on. (For you non-programmers out there, that's a little geek insider joke. Fsck is a unix command. It's short for File System ChecK.)

It quickly became evident that the root cause of the problem was some database corruption. The ad servers which actually served up the the ads would keep track of how many times a particular ad had been served and periodically dump those counts into a database. The biller would then come along and periodically collect all those counts, roll them up into an invoice, and bill the credit cards. The database was filled with entries containing essentially random numbers. No one had a clue how they got there.

I began the process of manually going through the database to clean up the bad entries, roll back the erroneous transactions, and send out apologetic emails to all the people who had been affected. Fortunately, there weren't a huge number of users back then, and I had caught the problem early enough that only a small number of them were affected. Still, it took several days to finally clean up the mess.

Now, it's a complete no-brainer that when something like that happens you add some code to detect the problem if it ever happens again, especially when you don't know why the problem happened in the first place. But I didn't. It's probably the single biggest professional mistake I've ever made. In my defense I can only say that I was under a lot of stress (more than I even realized at the time), but that's no excuse. I dropped the ball. And it was just pure dumb luck that the consequences were not more severe. If the problem had waited a year to crop up instead of a couple of weeks, or if I hadn't just happened to be there watching the biller window (both times!) when the problem cropped up Google could have had a serious public relations problem on its hands. As it happened, only a few dozen people were affected and we were able to undo the damage fairly easily.

You can probably guess what happened next. Yep. One week later. Same problem. This time I added a sanity check to the billing code and kicked myself black and blue for not thinking to do it earlier. At least the cleanup went a little faster this time because by now I had a lot of practice in what to do.

Stress.

And we still didn't know where the random numbers were coming from despite the fact that everyone on the ads team was trying to figure it out.

Into the breach

Ron sez:

I dove into the adstoo project with as much enthusiasm as I could muster, which I'm ashamed to say wasn't much. The situation was exacerbated by the fact that we had no Java development infrastructure. We were writing bareback, so to speak. We had no debugger. We were using JSP, but had no editor support for JSP syntax. (That turned into a real debugging nightmare. It could take many tens of minutes to find a single typographical error because the only indication that there was a problem was that the output just looked all wrong, but the actual problem could be very far away from the apparent source of the problem.)

Fortunately for me, I was assigned a junior engineer to work with/for me, and he actually knew what he was doing. While I struggled to learn the Java libraries and debugging techniques (I knew the basic language, but I had never done any serious development in it before) this guy just took the bull by the horns and pretty much just wrote the whole damn thing in a matter of weeks. I sometimes pull this old joke out of the dustbin, that in the ancient tradition of senior-junior relationships, he did all the work and I took all the credit.

That's not quite true. I did end up writing the credit card billing and accounting system, which is a nontrivial thing to get right. Fortunately for me, just before coming to Google I had taken some time to study computer security and cryptography, so I was actually well prepared for that particular task. Back in those days internal security was more or less nonexistent. All the engineers had root access to all of the servers. But I believe in planning ahead, and I anticipated the day when Google was not going to be a small company any more, and so I designed the billing system to be secure against even a dishonest employee with root access (which is not such an easy thing to do). I have no idea if they are still using my system, but if they are then I'd feel pretty confident that my credit card number was not going to get stolen.

But on the whole I was struggling, not just technically, but personally as well. The situation was exacerbated by my manager, who thought that the answer to my falling behind schedule was to start micromanaging me. That just made things worse. Much worse. I had been a senior scientist at a national research lab, essentially the equivalent of a tenured professor (but without the teaching responsibilities). I was used to being my own boss more or less, and I really resented being asked to make detailed lists of everything was going to do every single day. After all, Google had come asking me to join them, and so I felt I ought to be accorded more respect. But on the other hand I had to admit that I was not really performing, and so all the micromanagement was in some sense justified. I responded, I'm ashamed to admit, with some passive-aggressive head games, and so the situation deteriorated rapidly to the point where my manager and I were barely on speaking terms.

Things were made worse by the fact that I had been assigned an office mate who was also new to Google, and who was not part of the ads group. Most of the other ads group members were sharing offices (or cubicles) with other ads group members, and so I felt I wasn't really part of the club. On top of that, I was away from home and didn't really have a life up there in Northern California. The stress mounted. I started to get paranoid that I would get fired before reaching the one-year mark. I started experiencing stress-related health problems, some of which are still with me today. On more than one occasion I came that close to quitting. To this day I have no idea why I didn't.

It was about this time that I had my one and only meeting with Larry Page. It was to discuss the progress of the adstoo project and to set a launch date. My manager was there along with a couple of other people (including Doug I think). Things went smoothly until Larry suggested changing the way billing was handled. I don't remember the details, but my response was that this would be significant work. No one challenged me, but I found out later that the reaction of people in the room was something along the lines of, "Is he crazy? This ought to be a trivial change." This little incident turned out to have very far ranging repurcussions later, but that will have to wait for the next blog entry.

Somehow we actually managed to launch AdWords on schedule, in September of 2000. It still seems like a bloody miracle. Most of the credit goes to Jeremy, Ed and Schwim. It could not have been done without them.

I can still remember watching the very first ad roll in. It was for a company called Lively Lobsters. Two months ago, after five years of intending to do so, I finally bought myself a little toy stuffed lobster to commemorate the occasion. (Update on 12/9/2005: It appears that Lively Lobsters has gone out of business. There's some irony for you.)

About two weeks later all hell broke loose.

Day One

Ron sez:

My first day at Google went like this:

5:30 AM, wake up.
6:00 AM drive to Burbank airport.
6:30 AM stand in line at the Southwest airlines gate to get one of those now defunct plastic boarding passes
7:30 AM board the plane
9:00 AM or so land in San Jose
9:30 AM get the rental car
10:00 roll up to the Googleplex

It was a murderous commute. Four hours, which I eventually whittled down to three after a year of optimizing. I went through various permutations of cabs, rental cars, having my wife drive me to and from the airport (that didn't go over well at all) before I finally settled into something that vaguely resembled a routine. I ultimately ended up driving two different cars back and forth between LA and San Jose, and on a couple of occasions I did my commute in a Cessna 182RG (I'm a pilot) which was just too cool for words.

But getting to San Jose wasn't nearly as bad as trying to get back home. Originally I took Monday to be my telecommute day so that I could be at Google on Friday afternoons for the weekly TGIF meeting. TGIF is, as far as I know, still a Google tradition where everyone gets together to hear the latest developments from upper management (which back then meant hearing it directly from Larry and Sergey. Maybe it still does, I don't know.)

Trying to get on a Southwest flight at the San Jose airport on a Friday evening in mid-2000 was a serious nightmare. Heck, just trying to get to the airport was a nightmare. (I remember on one occasion the traffic was so gridlocked that I could not get to the rental car return, which back in those days was still in the main parking structure. I just left the car in a remote parking lot and walked from there to the terminal to make my flight. I guess the rental car company eventually found the car because only the normal rental charges ever showed up on my credit card.)

I found a place to stay, renting a room from Susan Wojcicki (now Google's director of product management) and her husband Dennis. Really cool people, and coincidentally, their house is also the one where Google first set up shop, so I really felt like I was in the bosom of history.

In the back of my mind there was one niggling little worry: I had neglected to ask Urs what I would actually be doing. Having decided to embark on this adventure I guess I decided it didn't matter. I was going to do something new and different, and, more to the point, I was going to learn how this incredible search technology worked. And I was going to knock Urs's red socks off with all the cool ideas I had for new features.

Things started off really well. The work environment at Google was everything that it has since been reputed to be. I learned to ride a unicycle while I was there, and my pool game improved considerably. My office had a spectacular view. Chef Charlie's cooking was delicious. (I still make some the recipes he gave me, but I can never get them to come out nearly as good as he did. I hear he's planning to open up his own restaurant soon. I'm looking forward to patronizing it.) The conference rooms were all color coded, except for the main one which was called the Lucky Lounge. Meetings, in stark contrast to my experience at JPL, started on time. It was all very refreshing and energizing. I was brimming with enthusiasm.

So I was a little disappointed when I found out on day 1 that I had been assigned to the ads group. But that disappointment turned to dismay when I learned what my assignment was to be: I was the lead engineer on a new advertising system code named "adstoo", what eventually became AdWords. That part wasn't so bad. The bad part was, this was going to be the inaugural Java project at Google. Google, which had until now been a Java-free zone (which was one of the reasons I took the job) was going for Java in a big way, and I, the consummate Java hater, was supposed to be its chief evangelist.

Just peachy.

On top of that there was trouble at home. My wife was having a really tough time trying to keep everything together back on the home front, what with the new house and four-legged kids and all. Our finances were starting to look a little rocky because I had taken a pay cut and taken on extra expenses for weekly plane tickets, rent, and a second car. (Google did give me a travel allowance, one of the factors in my decision to take the job, but my salary and travel allowance together were still less than my old salary had been at JPL.)

And then there were the stock options.

Yes, stock options can make you rich, but getting to that point can be pretty damned annoying. When you sign on to a company that has stock options as part of the compensation plan you get an option grant up front, but you don't actually vest any of those options until you've been at the company for a year. After that you vest a few more shares every month, but if you quit -- or get fired -- after eleven months and twenty nine days you're screwed.

Notwithstanding all that, you can actually choose to excercise your options up front even though they aren't vested, and it's often wise to do so because if you wait and the stock price goes up then when you do exercise the options then the difference between what you paid and what they were worth at the time counts as income for purposes of the alternative minimum tax despite the fact that you might be restricted from selling your shares because, for example, your company hadn't gone public yet. A lot of people got badly burned during the dotcom crash because they exercised their options when they couldn't sell the stock, generating a huge amount of AMT income on which they owed taxes. Then they watched helplessly as the stock price crashed, often to levels below what they had paid for the stock. So not only did they lose some or all of the actual cash they had paid to exercise the options, they also owed huge tax bills on paper "profits" that not only had they never actually earned, they could not have earned even if they wanted to because their stock was restricted. It was horribly unfair.

On the other hand, if you vest in your options but don't exercise them and then leave the company then the options expire and become worthless. Since I wasn't really planning to stay at Google for the long haul I figured it would be wise to exercise early to avoid the AMT trap. But to do that I had to come up with the cash to actually pay for the stock. And to add insult to injury, the day I joined Google they announced that they had finalized their deal with Yahoo, and the board raised the stock price by a factor of nearly 5! So not only did I have to come up with the cash, I had to come up with five times as much cash as I was expecting to have to come up with! My wife was not pleased.

In retrospect of course it all turned out OK, but at the time it was nerve-wracking. The dotcom crash was gathering steam, and Google's long-term prospects were far from clear. Still, growth was steady and there was electricity in the air, so I was fairly optimistic, even at the time. But writing that check was painful.

All in all, it was a stressful week.

Comment disez-vous "Xooglers" en Francais?

Throwing out a quick line to Franck Poisson, a former Googler who maintains a blog about the European search industry for those with a penchant for French. Once you read it, you'll be hooked. You'll read it to la fin. It's search related information on a grand scale. I can't salmon the strength to go on, but I think you get the bass-ic idea. Feel free to add your favorite bilingual puns on his name in the comments below. I'm pretty sure he's heard them all before.

Please allow me to introduce myself

Doug says: With Ron and I both posting at some length, it may be hard to tell who's talking at any given time. So we've agreed to identify ourselves at the beginning of each post to make it easier for you to know who's who without having to scroll all the way to the bottom. If/when other Xooglers join us, we'll ask them to do the same.

The wormgear turns

Ron sez:

It ocurred to me that by putting up my story piecemeal like this most people are gong to be reading it in reverse order. Hm. I wonder if there's a way to reorder postings -- or if people are so used to this that they tend to read blogs backwards anyway. Wow, the things you start thinking of.

Anyway...this is part three of my ongoing saga of my year at Google.

Of all the random shit that has affected my life surely none has had more impact than the fact that Google would not take no for an answer. They asked if they could fly me up again to try to change my mind. And they wanted me to bring my wife along as well. Well, heck, a free dinner for two, how could I turn that down?

The person assigned to win me over was Urs Hoelzle, at the time VP of Engineering. Three things impressed me about him. First, he told a really good story about how Google was planning to make money (one that turned out to be pretty much right on the money, so to speak). Second, his technical background was in programming languges, which was a big plus for me for reasons that I will explain shortly. And third, he had a really huge dog that he brought to work with him. (Yoshka, the Leonberger, has since become the stuff of legend.)

(Another interesting bit of trivia about Urs: he always wore red socks. Something to do with him being Swiss, I believe, but I never fully grokked it.)

To fully appreciate the horrible irony of what happened next, I have to tell you about this peculiar affectation (or perhaps the word I'm lookin for is affliction?) of mine: I am a fan of Lisp (the programming language, not the speech impediment).

For those of you reading this who are not programmers I have to digress even further and explain a little about programming langauges. Programmers are craftsmen, and craftsmen can sometimes get a little weird about their tools. It's funny to watch Tim Allen wax rhapsodic about the relative merits of the Binford model 2330 frabnobulator because, like all good comedy, it contains more than a grain of truth. Programmers are craftsmen, and programming languages are their primary tools, and so it is not uncommon (understatement of the day) to see spirited discussion of the relative merits of this or that language.

One of the reasons that I had chosen to stay safely cloistered in the ivory tower of JPL for over a decade was that my favorite tool/language had definitively lost the language wars. I got into Lisp in the 80's because it was the language used for artificial intelligence (AI) work, which was the field I wanted to get into. Unfortunately, AI failed to deliver on lofty promises, and so it fell out of favor with its (mostly government) sponsors. As a result, industry abandoned Lisp in favor of C, and later, C++, Java and Perl.

I don't want to rehash the language wars here. Suffice it to say that I attributed a significant portion of my own professional success to my use of Lisp over the years, and I was loath to give it up. But the industry had moved definitively away from Lisp and towards C++ and, in the Web world, Java. I found both of these languages very difficult and frustrating to master, and that was a large factor in my decision to stay at JPL during most of the dotcom boom. Why should I submit myself to all that pain when I had a perfectly good job where I could do as I pleased?

But in Urs Hoeltzle I saw a beacon of hope. I had done my homework on Urs and found that he had made a major portion of his career in the development of a language called Smalltalk, which was very similar to Lisp in many important ways. In particular, it was not a mainstream language, so I thought that in Urs I would find a kindred spirit, an engineer who knew what it was like to work on and perhaps even fall in love with a programming language that was not part of the mainstream.

My wife and I have different recollections about what happened next. I have very strong memories of my telling her about starting to change my mind about Google, and her responding that she wasn't too enthusiastic about the prospect, but that if it was what I wanted to do that she would support me. She insists that she never said any such thing.

It seemed like a good time to leave JPL. Things were changing there and I was beginning to get somewhat marginalized on the project I was working on. (That's a whole 'nuther long story. If you're interested you can read it here. Be aware that it contains some spoilers for the Google story, so if you're hanging on to the edge of your seat waiting for the next installment here at xooglers (yeah, right) you might want to give it a miss.) So I called Google and we set a start date in June of 2000, and gave notice at JPL.

To my astonishment, people started coming out of the woodwork to express their shock and dismay that I was leaving JPL. Several people who I was sure would be happy to see me leave actually asked me to reconsider. It did my ego a lot of good to hear it, but it was too late. I was off to Mountain View.

Smart enough to know better

Ron nails it when he talks about feeling intellectually inferior in the Googleplex. In my case, that might be because it was an accurate assessment of my position in the educational hierarchy (my only degree was a lowly BA -- in English, no less), but everyone I talked to while I worked there seemed to feel like they were pulling down the curve.

One of the things I admired about Google was its lack of intellectual snobbery. That will sound like absolute bullshit to a lot of people, who are sick of hearing about how smart Googlers are and how hard it is to get a job there unless you've evolved into some sort of higher level human with a brain the size of Beldar. So let me clarify: there was no intellectual snobbery at Google -- toward other Googlers.

Google truly developed a meritocracy of ideas. The benefit of a hiring process that pokes, prods and sniffs applicants in ways that only Dick Cheney could appreciate, is that once you're in the door, it's assumed you're the real deal. So anyone could, and usually did, comment on any aspect of any project that they found worth examining.

I'll give two quick examples.

I remember a meeting in Larry and Sergey's office attended by a half dozen Googlers from engineering and marketing. One of the people there was a new assistant product manager who I suspect hadn't even returned his Stanford graduation gown yet. When the meeting broke up, I hung around to harass Sergey about some marketing question he had been avoiding. Time like this was precious because it was often the only way to force a decision on issues that weren't key to keeping the site up and running.

As the group filed out, I started making my case to Sergey, expecting to have five minutes mano-a-mano in which to persuade him. I was surprised when he looked over my shoulder at Nikhil, whose curiosity had caused him to linger, and asked him his opinion of my argument, then listened carefully as Nikhil laid out a cogent, well-argued response.

I confess, I wasn't happy that this... this... whippersnapper with no experience at Google was being asked to sit in judgment of my proposal. Didn't he realize he was a junior staff member and shouldn't be hanging around this conversation to begin with? This would never have happened at the Good ol' Merc, where proprieties of rank were carefully observed and it would be unseemly, impolitic and career-threatening to blatantly refute a manager in front of the company's top executive. And dammit, he was poking holes in my idea left, right and center, which Sergey duly noted.

After mainlining my blood pressure medication and walking a few times around Mountain View, I came to realize that Nikhil had some valid points. I also came to realize that I shouldn't have been surprised. It didn't matter that he'd only been on the job a short while. He was incredibly smart and he required no long experience in the Google milieu to construct a logical argument when asked to do so.

The second example occurred in a similar way. I had been working with a cross-divisional team on an extremely complex project for several weeks. After 20 or 30 rounds of revisions, we had carefully constructed language that met the requirements of legal, sales, marketing and business development and were presenting the final version to Larry and Sergey for approval. As usual, an LSA hovered about taking notes. The four LSAs (Larry Sergey admins) were responsible for keeping L/S on schedule and on track; without exaggeration, one of the most challenging, all-consuming positions in the company. I believe all were recent Stanford grads and of necessity, bright, articulate and extremely diplomatic.

Larry and Sergey hated what we had written and pretty much sliced all the way through our carefully constructed Gordian Knot. Then they began rewriting it from scratch, with a lot of chuckling, sarcasm and broad generalizations that ignored the subtleties we had struggled with so mightily. Larry actively engaged the LSA in this process and she made several suggestions he incorporated.

Again, I shouldn't have been surprised. Larry and Sergey almost always felt that more viewpoints meant better decisions. It's fundamental to the way Google itself works and they applied the principle across the board. As long as the individual voices don't coalesce into cacophony, value can be harvested from each perspective for the overall improvement of any project. And if someone were an LSA or a VP of engineering didn't really matter since no technical knowledge was required in this instance. The LSA had the intelligence and insight to contribute and she did so.

So, while Google's arrogance ultimately became a nasty undertone in the conversations about the company taking place in the press and among technorati trying to do business with us, it did not manifest itself amongst Googlers themselves in the early days. Given my status as a marketeer and a liberal arts major, I likely would have been well-positioned to see it if it existed.

That being said, I'll come back to the topic in future posts to explore how Google interacted with those outside the company, and with those who became part of the AdWords empire within it. Their tales didn't always have such storybook endings.

In the beginning...

The memory of my first encounter with Google is seared into my memory like JFK's assassination is supposed to be seared into the minds of many baby boomers.

As an aside, It's always seemed odd to me that the baby boom generation is supposed to include anyone born through 1964. Technically I'm a boomer, but JFK was long dead when I entered first grade. On the other hand, I do clearly remember what the world was like before there was Google, before there was the World Wide Web, and even before there were personal computers. I am a member of the last generation to know what that world was like. God, do I feel old now.

But I digress.

I was reading a usenet newsgroup (I'll tell you which one later -- it bears on a peculiar personality trait of mine that has some bearing on my Google experience later on) when someone answered a particularly obscure question and followed up with "Thank God for Google" or some such comment. (When Google acquired Deja and brought their database of old usenet postings on line I went looking for the post that started the whole thing and couldn't find it. This was just the first of many Twilight-Zonish (or X-Filesish for you younger readers) events that happened to me of the course of the next few years.)

Google? What's Google, I wondered. So I pulled up a web browser and took a wild guess (which was as good a way as any of finding things on the web in those days): www.google.com. Oh, it's a search engine, kind of like Alta Vista. But, holy shit, it's fast! And it has this uncanny way of putting just what you're looking for right at the top of the results list.

To understand what happened next I have to give you a little background about my professional life up to that point. I had moved to California from Virginia in 1988 along with my Ph.D. thesis advisor, David Miller, who had just been offered a job at JPL to do research for the Mars Rover program. Over the course of the next twelve years my career had a lot of ups and downs, and I had a lot of opportunities to leave JPL, but somehow the opportunities always seemed to come along at the "up" times when life seemed good and I wasn't much inclined to rip up my roots, which were growing deeper as the years went by.

But the day I found Google just happened to be during a "down" time.

My first thought was, "How the hell do they do that?" Alta Vista was astonishing enough in its day, but this took speed and accuracy to a whole new level. I had always been idly curious about how Alta Vista worked, and now I just had to know. In a fit of what Alan Greenspan would have called "irrational exuberance", I dashed off a resume.

Google got back to me with astonishing speed. It was early 2000, the dotcom bubble was just reaching its peak, it was a seller's market when it came to any kind of technical talent, and I looked pretty good on paper. If memory serves, it was only about two or three hours before my phone rang. A week later I was flying up for an interview.

I don't have nearly as many colorful interview stories to tell as Doug did. I met with half a dozen people. (Neither Larry nor Sergey interviewed me.) They grilled me on the usual things -- caches, hash tables, virtual functions, etc. It was a pretty standard technical interview as best I can recall, with a few Googley twists (how would you write a program that could identify news sites on the web?) I guess I must have hornswoggled them pretty good because they made me an offer. (That may sound like a bit of self-deprecating humor, but it isn't. The truth is I really wasn't qualified for the job. But that didn't become apparent until later.)

So now I'm in a pickle. On the one hand I've got this job offer and an opportunity to learn how this cool technology works, and maybe even make a few bucks on the stock options (though that was never the main motivation for going. It was pretty clear even in early 2000 that the internet bubble was gong to burst sooner or later, and besides, how was a search engine ever going to make money?) More to the point, I was worried that if I didn't get away from JPL now I never would, and I didn't really want to retire without ever having experienced anything but working for one organization.

On the other hand, I had a pretty cushy situation. My seniority at JPL made it possible for me to work on pretty much anything I wanted to. I was well paid (by my standards at the time). My job was (or seemed) secure. My wife and I had just bought a nice new house and gotten a dog and a cat. (The Southern California real estate market was just starting to pull out of a slump and we were able to buy for what now seems dirt cheap.) To take this job we would either have to move or I would be doing the commute from hell. Neither of those prospects seemed very appealing.

I made my decision while visiting some friends in upstate New York. One of my friends had just had her father die unexpectedly. He was an orthopedic surgeon. Very wealthy, or so it seemed to me at the time. Full of life. Commuted from his farm in Virginia to his job in his own Bell Jet Ranger. (I remember he flew in one night while I was visiting my friend on the farm. He was a real regular down-to-earth guy. Asked me if I'd like to join him tomorrow -- he was going to the Pittsburg Steeler's summer training camp. I'm not that much of a football fan, but it was pretty damn cool anyway. There were fans everywhere wondering who the hell we were that we got to go into all the VIP areas.)

One day he just keeled over while skiing. Heart attack. He was dead before they got him off the mountain.

It was during that reunion that I had an epiphany: life is short, and I had everything I ever wanted: a nice house, a secure job that I (mostly) enjoyed, why in the world would I want to put myself through hell just to hang out at some dotcom that would probably be bankrupt in a year or two?

When we got back from New York I called Google and told them that I had decided to decline their offer.

Hello, world

The first post is always the hardest.

I've been debating with myself whether or not to write about my Google experience at all. It seems like such a self-indulgent thing to do, and, while in retrospect it has certainly turned out quite well, there were some bumps along the way (to put it mildly) and I did some things that I'm not altogether proud of.

On the other hand, I've always enjoyed being on the receiving end of a good inside scoop, so now that I'm in a position to share an experience that others seem to care about I feel like I ought to do so. Besides, they say confession is good for the soul.

A very brief summary to set the stage: I worked as a robotics and artificial intelligence researcher at the Jet Propulsion Lab from 1988 until 2000 when I went to work for Google as a software engineer. I was there for just over a year. I worked on two main projects, the first release of AdWords, and a little known widget called the Translation Console. If you go to Google's preferences page you will see that Google is available in well over 100 languages, including Klingon and Elmer Fudd. (There used to be a Swedish Chef option, but that seems to have succumbed to political correctness.) All those translations are provided by volunteers around the world. The interface they use to do the translating is the translation console (officially called the Google In Your Language Program). As far as I can tell it hasn't changed much since I wrote it. I'm pretty proud of that.

I guess the #1 FAQ for people who have left Google is why did you leave. My main reason for leaving was that I was commuting from Los Angeles. I'd fly up on Southwest early Monday morning, fly back on Thursday evening, and telecommute on Fridays and weekends. That regimen was pretty stressful even under the best of circumstances, but when 9/11 happened it became completely untenable. I had already given my notice before 9/11, but I don't think I could have stayed on after that even if I had wanted to. But I'm getting ahead of myself.

As I've said, my story is not entirely rosy, so I want to make a sort of blanket disclaimer, and to do that I have to indulge in a little bit of tooting of my own horn. I have a Ph.D. in computer science. I was a senior scientist at JPL when I went to Google, and when I returned to JPL I was promoted to Principal, the highest rung on their technical career ladder. (Actually, it turns out there are higher rungs, but their existence in not publicly known.) I am generally considered to be a pretty bright guy.

I am tooting my horn to put the following assessment in perspective: at Googe, if I were to rate people on general smartness I would have put myself in the bottom 25%. It was pretty much the first time in my life that I found myself not at the top of the intellectual pecking order. It was not an easy adjustment for me. But I'm getting ahead of myself again.

I wanted to say this up front because since the IPO there has been a steady chorus of criticism along the lines of, "Google has little real value and it's only a matter of time before it all comes crashing down like a bad flashback to February 2000." In my humble but better-informed-than-most opinion this is all sour grapes. Google is a valuable company because the people who built it are incredibly smart and they work incredibly hard. I feel priveleged to have been a (small) part of it. There are many valid criticisms of Google (and I expect I'll be making some of my own), but that they have built little of real value is not among them.

I'm not saying this because I want to kiss up to Sergey (as one commentor suggested might be the motivation for one of Doug's postings). I have no need to kiss up to anyone any more (and, though I don't have any firsthand knowledge, I strongly suspect that Doug doesn't either). I'm saying it because Google has taken a lot of bashing Some of it was well deserved IMO (like when Google blacklisted CNet for doing a story that included personal information on CEO Eric Schmidt which was obtained by doing a Google search), but most of it was (and is) not, and I just wanted to stand up and say so.

With that out of the way, I'll get on with the story.

Enough about me...

In my first post here, I invited other Xooglers to join me in reminiscing about life back in the day. Ron Garret has accepted that invitation. Ron was the lead engineer on the first release of AdWords, and the experience affected him so fundamentally that even his name changed.

It was great having Ron on the staff for many reasons, not the least of which was that he nicely filled out our hyperbole portfolio. We'd been talking about how smart our technology was and after Ron joined us from the Jet Propulsion Lab, we could truthfully say that, "Yes, Google is so complex that we have both a brain surgeon and a rocket scientist working on it."

Welcome Ron...

Has it come to this?

I think I may need to set some new ground rules for comments. I don't mind people disagreeing with Google policies, its technology or its stock price and venting their spleens here, though they might be better served doing so directly to Google via email or Google's own group.

I'm less enamored of obscenity-laden speculation on my ulterior motives, but since most of those posts speak more eloquently about their authors than about me, I have no problem leaving them on the site.

However, I need to draw the line at the use of comments to launch ad hominem attacks on individual Google employees, past or present. If you're looking to settle a score or make unfounded accusations, look elsewhere. Call me censorious, but if such comments are posted here, I reserve the right to remove them in their entirety. I'm prepared to take the heat for doing so.

After all, finally, have we no sense of decency?

AdSenseless

Well... it just goes to show how quickly I've forgotten all I once put into practice at Google. In my second post here, I closed with, "If you care, read on. If not, be sure to click an ad on the way out. Someone's gotta pay for these pixels." As a number of astute readers have pointed out, that's a likely violation of Adsense rules, which state, "You shall not, and shall not authorize or encourage any third party to: (i) directly or indirectly generate queries, Referral Events, or impressions of or clicks on any Ad..." Whoops.

Damn that Google and its high-handed dictates.

So I hearby renounce my directive to click an ad on your way out. Don't do it unless the ad is entirely relevant to you and so compelling you're unable to resist. Even then, think twice about whether you're ready to buy whatever they're selling. Google advertisers have been good to me, and I want to return the favor to them.

Speaking of Google advertising, the branding of the program in 2000 was an interesting challenge. We debated a number of names, with "BuyWords" and "AdsDirect" rising to the top among a list that included "PrestoAds" and "AdsToo." The keyword-targeting concept was still relatively new and we wanted a name that would give people some idea of how it worked.

As the email debated waged on, I threw out "AdWords," as a compromise. It seemed to meet the requirements and, as I pointed out to the ads team, it had a comforting familiarity about it. "AdWords." "Edwards." Hmmm...

We checked with the owner of the BuyWords domain and he wouldn't sell and we agreed that AdsDirect was too much like direct mail. So, AdWords it was.

And that's how my name came to be enshrined in Google's revenue stream.

Meet the new boss...

Here in handy reference form, is a quick comparison of my two most recent employers (I vaguely recall another Googler posting on a similar topic once). While the differences are great, there are some surprising similarities.

The Mercury News (1999)		Google (1999)
150 years as a profitable operation		1 year as a non-profitable operation
Thousands of empoyees, many with more than 50 years of service		Tens of employees, most with fewer than 3 months of service
Well-defined roles, with 7 unions to enforce them		No defined roles and strange looks if you ask about them
Key decisions made around an imposing boardroom table by a committee with the publisher presiding		Key decisions made in the cafeteria line while a founder is loading his plate with baked organic tofu
All new products based on P&L projections for five years out		Most new products based on an engineer developing something Larry or Sergey thinks is cool
Products not released until perfect - this is the first draft of history and the newspaper cannot appear fallible		Products released as soon as they're checked for security and stability. We'll let users tell us what needs to be fixed
Smart, articulate journalists, who know what people really need, even if they don't		Smart, articulate engineers, who know what people really need, even if they don't
No tolerance for marketing, which is an unfotunate necessity, but taints the journalistic mission		No tolerance for marketing, which is an unfortunate necessity, but taints the engineering mission

Here's your desk. Now start doing marketing stuff.

So, now I was Google's online brand manager. What exactly did that mean? I didn't have a clue, and evidently no one else did either. It was as if some corporate biological alarm clock had gone off: "You know, we're at that point where we need to have somebody to do all that stuff that's not engineering. Let's get us some of them marketing folks. And since the world is divided between online and offline, we'll get one of each."

Initially, Google's marketing department consisted of a small cadre of staffers who had very mixed experience in the field. Cindy was the acting VP and had been recruited out of Apple's P.R. department, where she had built a reputation as an astute professional who could talk tech with the big guys. Susan was a Harvard grad with an MBA and had worked in marketing at Intel. It was in Susan's garage that Google set up its first post-Stanford office. My counterpart, an offline brand manager who started the same week as I did, had worked extensively for traditional packaged goods companies.

The others in our group were new to marketing. Google tended to hire smart Stanford grads and set them loose in the halls. If they didn't secure a role elsewhere, they ended up in marketing, where the assumption seemed to be that no special skills were required, as long as you could think quickly and were willing to work hard.

That was pretty much true across the company. While engineers needed technical skills, even there Google didn't value experience in a particular field as much as a track record of intelligence and accomplishment. This flowed directly from Larry and Sergey's confidence in their own ability to figure out anything they put their minds to. Smart people should be able to learn whatever they needed and then implement it. L/S had initially handled all aspects of the company, after all, from UI design to answering user email, so they knew it wasn't that hard.

To understand Google's hiring policies and organizational structure, it helps to think of employees as cells within the corporate corpus. It's useful to have cells that serve specific functions when the need arises, but it's inefficient to have those cells hanging around sucking energy from the rest of the organism if their singular function is no longer required. Better to have cells that can adapt themselves to any situation, solve the problem and then move on to the next issue. Yeah, we're talking about stem cells.

Googlers should apply themselves to any project that needs doing, then take on totally unrelated tasks without hesitation. Moreover, they should be able to identify those needs on their own and teach themselves how to solve the problems they present. That's why the company is so fixated on hiring only really smart people without much regard for their prior work experience (no need to comment this post asking how I slipped through).

So Jim Reese, a neurosurgeon became head of operations. Courtney, who was hired as a market researcher, became a media buyer, Dennis, the assistant webmaster, took on logo designs and search quality for our Korean site. I took on responsibility for building out customer service and internationalization, neither of which I knew anything about.

It wasn't easy for me to adapt to this mindset, as I'll explain in a future post, and I almost didn't make it. I suffered from what Sergey would often describe as a "big company attitude" that led me to expect rules and process and clearly defined areas of responsibility. In a company of 60 people with millions of customers and a growth curve like, well, like Google, there wasn't room for specialists or walls around a person's role. Your job was whatever the day demanded and if you needed to learn HTML or Swahili to do it, that was your job, too. It was hard and it was frustrating and it was pretty much the only kind of job anyone should ever want to do. I can't imagine any Googler from that time joining a big company and getting the same satisfaction. Once you've been a stem cell, you're just not going to be happy spending the rest of your existence as a blood vessel and lymphatic vascular endothelial fenestrated cell.

Let's talk turkey

Tomorrow is Thanksgiving, which means right about now, Dennis Hwang is likely squinting at his tablet PC desperately trying to come up with a creative way to incorporate a turkey into the Google logo yet again.

Dennis is a wonder and one of the people who has quietly instilled Google's brand with it's unique flavor. He has gained international acclaim for the logos he creates, but few people realize that he works alone and does the logos as a sideline to his real job of keeping a raft of international Google websites afloat. That means Dennis, who only sleeps between 6 and 9AM, is usually cramming against a deadline to get the logo done and approved hours before it posts.

The process, when I was part of it, was highly efficient and extremely stressful. We'd set up meetings quarterly to map out a calendar, discuss which countries would be offended if we left out their biggest holiday, throw out suggestions for people and events worth commemorating and make sure we were not becoming predictable in what we selected to put on the home page. Then, for some reason, the meeting would be cancelled, rescheduled and postponed until someone would note that, "Oh my god, tomorrow is Mother's Day! What are we doing for a logo?" And Dennis would go to work, churning out one great idea after another within the hour.

There was a lot of back and forth on logos, with the core group of Dennis, Karen, Marissa Mayer and myself debating the merits of keeping dancers on their toes by celebrating George Balanchine's 100th birthday vs. pissing off a billion Chinese Internet users by ignoring their New Year's celebration. Most logos were not terribly contentious, but because design involves aesthetic judgment, everyone had an opinion when Dennis sent around a link to his artwork. (No one at Google sends attachments via email; everything is posted on the intranet. Many Nooglers find out the hard way that sending files to large groups is flame bait.)

Sergey provided the final sign off and he always encouraged Dennis to push the envelope a bit. Every once and a while things went off the rails. There was the time one of Dennis' Olympians displayed an unfortunate tenting in his toga and we received about a dozen emails asking why the Greek Guy on our homepage had a boner. And the time Dennis' Shichi-Go-San logo for Japan featuring a crane, a turtle and a traditional candy bag was read by many non-Japanese users as a turkey, a turtle and a thermometer.

I'll see if I can get Dennis to add his favorite stories, but he's an amazingly self-effacing guy and always overbooked, so I'm not optimistic. The story of how the logos came to be at all is also worth telling and I'll return to that in a future post.

You can see the history of Google turkey logos here, starting with Sergey's first clip-art turkey from 1998.

Room to grow

Jim Reese took these shots of Google shortly after the move to 2400 Bayshore Parkway from downtown Mt. View. Yes, the desks were wooden doors mounted on metal sawhorses. They had two ergonomically correct settings: "nose bleed" and "knee knocker." No expense was spared in the early days, a tradition that never really changed much.

Answer a hard question, get some raw fish

Sergey seemed satisfied with my answers about viral marketing and rather pleased when I told him that I didn't think a large marketing budget would be a good idea for a company in its early stages. I thought shooting hamsters out of a cannon on national TV was a waste of money. Having worked at a PBS affiliate and a newspaper, I was used to doing a lot with no budget.

He was friendly and relaxed and didn't laugh at the scraps of the conversational Russian I was able to summon when he asked about my time in Siberia. Finally, he leaned forward for the big test.

"I'm going to give you five minutes," he told me. "When I come back, I want you to explain to me something complicated that I don't already know." He then rolled out of the room toward the snack area. I looked at Cindy. "He's very curious about everything," she told me. "You can talk about a hobby, something technical, whatever you want. Just make sure it's something you really understand well."

I reached for a piece of scrap paper as my mind raced. What complicated thing did I know well enough to describe to Sergey? Diaper-changing didn't seem appropriate, though it took a long time to master. The process for getting one's wife to go see the Matrix instead of a chick flick was probably too complex, even for him. I thought about explaining how newspapers are printed, but decided to go with the general theory of marketing. Fortunately, it was fresh in my mind, because I'd only learned it recently.

The deep dark secret of my career is that I managed to get by without ever taking a single course in business or economics. I did take Planetary Geology, Latin and Spenserian Verse on my way to a degree in English, but my boss at the Mercury News had a Harvard MBA and a desire to drive some common business theory into my thick skull. She gave me a bunch of her old textbooks and suggested I spend some time reading them. I'd found a couple of them interesting, including Porter's Competitive Strategy and Aaker's books on branding.

I began regurgitating everything I could remember onto the paper in front of me: The five P's (or was it six?), the four M's, barriers to entry, differentiation on quality or price. By the time Sergey came back, I had enough to talk for ten minutes and was confident I could fill any holes with the three B's (Buckets of Baffling Bullshit). Sergey paid real attention as I went to the whiteboard and drew circles and squares and lots of arrows. I found out later that he asked almost everyone to do this, so if a candidate wasn't hired, at least it wasn't a total waste of his time.

It was getting dark by the time I finished and Sergey invited me to join the staff for dinner, which was being brought into a small kitchen/snack area across from the conference room. A crowd of hungry engineers bounced from plate to plate with chopsticks picking at a large selection of sushi.

"We just hired a chef, so this is a temporary setup," Sergey told me. "And we've got two massage therapists coming in as well."

An alarm bell went off in my head at that. This was the guy who didn't think there should be a marketing budget, and he had hired a chef and massage therapists? But then I saw the platters of fatty tuna and shrimp and salmon and yellowtail. I grabbed some chopsticks and began loading my plate. Concerns about a business plan and revenue streams and organizational structure faded away. I could always bail for the next startup when Google ran out of money. And in the meantime, I'd eat well and be really relaxed.

Two weeks later, I started as Online Brand Manager.

My dinner with Sergey

A couple of days after my first interview, I got another call. Could I come back to meet some more of the staff? I could and I did. I chatted with Scott Epstein, the interim VP of Sales who was phasing out after putting forth a plan to spend millions on an advertising campaign, an idea that didn't go over terribly well with Larry and Sergey. He wished me well. I met with Urs Hoelzle, Google's head of engineering and owner of Yoshka, a free-range wooly mammoth that Urs assured me was perfectly harmless, as long as I didn't lie down on the floor and act like a chew toy. I met with Omid, the newly hired head of sales, who was so genuine I felt safe confiding in him that I was not a big fan of AOL, only to discover that he was a Netscapee who had been at the company when it was acquired by AOL.

And then Cindy brought me back to the conference room in which I'd begun to wait for Sergey Brin. I wasn't nervous. I knew that Sergey was about the age of my favorite t-shirt and was Russian by birth. I had been to Russia. I knew Russian. Russians were friends of mine. I understood their dark humor, their cynical views and their sarcastic ways. It was one of those rare times when I felt a sense of confidence going into an interview that it was all going to work out just fine. I could handle this guy. If I ended up working with him, I could probably mentor him on how to run a successful business as we toasted success with Siberian vodka.

When Sergey showed up, my initial impression was even more reassuring. He was wearing gym shorts, a tee shirt and inline skates. He had obviously been playing hard. I'd known better than to wear a tie, but he took office casual to a new level. I sat back and began toying with one of the rubber balls, feeling so relaxed that I accidentally removed its stopper, causing half the air inside to rush out with a hiss. Sergey seemed to find that amusing. He quickly pored over my resume, and began peppering me with questions. "What kind of marketing did you do that was most effective?" "What metrics did you use to measure it?" "What types of viral marketing did you do?" "What was your GPA?" I was doing fine until that last one. I just looked at him.

"My GPA?" I asked. I hadn't thought about my GPA since the day they handed me my diploma in 1981. And given that Brown allowed me to take as many classes as I wanted with a pass/fail option, I'm not sure I ever knew what my GPA was. I laughed it off, thinking he was joking, but even after I had an offer on the table, the HR people kept pestering me for a college transcript and my S.A.T. scores. It was a classic Google moment. Your S.A.T. score was the measure of your intellectual capability; your GPA represented the numerical summary of your ability to execute on that potential. Your value to Google could be plotted using those two data points.

Sergey's desire to reduce every decision to an equation would cause me a fair amount of frustration in the years to come. While it forced a discipline on me that was likely lacking in my career up to that point, it also went against my deeply-held conviction that some things are not expressible simply by deriving the correct algorithm. A lot of engineers at Google would dispute that with religious conviction, though they might admit that deriving the correct algorithm would be "non-trivial."

But that was not Sergey's hard question. He saved that for last.

Where to begin

So how does one interview for a job at a startup in Silicon Valley? I'd had lots of practice by the time I pulled into the Google parking lot. It was another warm Bay Area November and I wasn't terribly surprised to see one section of the parking area roped off with police tape and a roller hockey net at each end. The squat beige building and dozens just like it grazed on verdant fields of rolling landscaped lawns interspersed with tasteful fountains and ambiguous sculptures. When I entered the first floor of the building, there were arrows printed on copy paper pointing the way to the stairs, which I followed to the second floor.

A young, curly-haired receptionist walked me to a small conference room decorated with a nine-foot whiteboard, a standard issue circular table and several large inflatable rubber balls of various sizes. A squirrel ran along the ledge outside the window and I could see delivery men moving another dot com in across the way. As I sat idly patting a three-foot ball, I was introduced to a number of folks who seemed to have some connection with marketing.

Susan Wojcicki and Cindy McCaffrey walked me through a general introduction to the company with the sort of positive energy and optimistic outlook that was everywhere in those days. They had recently won recognition from Time magazine, traffic to the website was growing by leaps and bounds and they had strong financial backing, though no immediate source of revenue. That would come in time they assured me. They asked about my experience, especially with viral marketing. Cindy indicated that was important to the company's founders. I gave the usual interview song and dance, flashed some samples from my portfolio, and assured them that it sounded like a great opportunity. We shook hands and I headed back to my car.

It’s tempting at this point to claim that as I crept back along Highway 101, I analyzed what I had heard, cross-referenced it with my mental map of the Internet's future and realized that this was the real deal; that this would be the one startup in all of Silicon Valley that didn’t flop on its side, thrash its tail, and float belly side up when the great dot com wave rolled back from the crest of NASDAQ to the dusty shores of Lake Lagunita. I didn't realize that at all. Mostly I realized that large rubber balls were a strange way to decorate an office.

My next visit would reveal some of Google's even stranger ways.

Google goes electric

There's a lot in the news these days about Google Print... er, Google Book Search. People wonder how Google could so arrogantly assume that they can scan books that are under copyright and offer up snippets to online searchers. I'm not going into the details of the discussions that led to that decision here, but I'd like to offer my own observations on the culture that bred it.

First, I accept that Larry and Sergey really are brilliant. I'm sure that on IQ tests, they're off the charts, but that's not the kind of brilliance I mean. I mean brilliant in the sense that they have a vision that burns so brightly within them it scorches everything that stands in its way. The truth is so obvious to them that they have no patience for the niceties of polite society when bringing that vision to life.

Obviously, it's a good idea to make as much information as possible available to as many people as possible. Obviously, a lot of valuable information is in books. Obviously, helping people find that information is good. Obviously, an author only benefits if people find out that his or her book contains useful information. There are no shades of grey in this. Truth is, after all, a binary function.

It's not important to slow down and explain things, if the value of what you're doing is so self-evident that eventually people will see it for themselves. You'll hear a lot on this topic from me, because I think it's Google's core strength and it's greatest Achille's heel. It has played out time and again in the Google story, from the earliest launch of a better search engine in a crowded field, to the firestorms around Gmail and now Google Print.

Recently, I heard a recording of Bob Dylan at his 1966 Royal Albert Hall Concert. Halfway through the show he switches from an acoustic solo performance to an electric guitar backed by a full band. The reaction of his folkie fans is not positive. At the end of "Ballad of a Thin Man," a crowd member yells, "Judas!," to which Dylan replies, "I don't believe you!" A pause. Then with disbelief and anger twisting his voice, "You're a liar!" He then tells his band to "play fuckin' loud" and rips into a raw and wrathful "Like a Rolling Stone."

The first time I heard that, I thought to myself, that's Larry Page. It doesn't matter what the fans think, when it's so obviously the right thing to do, you just do it. Eventually, they'll understand. Even if that means for a while, you're out there on your own....

Moving On

I'm beginning to understand why this blog thing is taking off. There's no editor to tell you to shut up and get on with it. There's no limit to the number of words you can spill into the ether like Onan's seed with equally productive results. At this point, I don't even think anyone's reading this, so I can parade around naked prose with no self-restraint. As my Google co-workers can attest, I keep typing til all the white space is gone.

And (at last), speaking of Google...

It was in November of 1999 that I began interviewing with Google. I'd been at the San Jose Mercury News (a Knight Ridder newspaper) long enough to earn a pension and I was anxious to move on to a company entirely in the Internet space; maybe a vertical hub or a vortal or clicks-and-mortar etailer. The Merc had done some good things with online and actually been a pioneer among newspapers, largely thanks to Bob Ingle, a visionary old-school editor who set up Mercury Center, the paper’s online division. Under Ingle’s firm hand and frequent four-letter imprecations, Mercury Center took on the role of Knight Ridder’s real-time experiment in electronic circulation. I got to pull together a business plan for SiliconValley.com and see it launched, but mostly I played around the edges while really smart guys like Barry Parr figured it all out and explained it to me.

But, hey, this was 1999. Silicon Valley was awash in startups. They were in trendy former warehouses in San Francisco’s Multi-media Gulch. They were in clusters of rented closets that shared utilities and blackened Mr. Coffee machines. They were in shiny monuments to excess along Highway 101 with purple walls and built-in slides leading to their lobbies from their upper floors. They were in the craniums of every man, woman and child with an ounce of entrepreneurial spirit between Gilroy and Petaluma, cooking up like idea popcorn. Most died quietly; half-baked, burned out, warmed over, unpalatable. But occasionally one would explode into a wildly successful business and the Valley would come running, throwing resumes and venture capital at the new marvel of fluff and air.

I wanted a piece of that.

I don't remember exactly how I first learned about Google. It may have been because we shared a vendor who built company stores. I was trying to turn SiliconValley.com into the corporate cultural heart of the Valley and invited local tech companies to join our online mall for branded premiums. While checking out Google's online store, I noticed they were hiring. I'll admit it; at that point I would have talked to anyone who had a business plan with "Internet" scrawled in crayon across the top. Mercury News editor Jerry Ceppos had identified the period as “the equivalent of the Italian Renaissance, happening right in our backyard.” And while the Merc was one of the best places in the Valley from which to observe the emerging e-Medicis and dot-Botticellis, I wanted to get my hands on a slab of marble and a chisel and make some dust of my own.
My research into Google uncovered these telling facts:

• Google was an Internet company founded by a couple of guys from Stanford.
• Google had funding from two venture capitalist firms known for giving cash to anyone who could spell U-R-L.
• Google’s sole product was a unique new search engine, kind of like Yahoo, Lycos, Excite, Ask Jeeves, Direct Hit, Altavista, Fast, Dogpile, Webcrawler, HotBot, Northern Light, and InfoSeek -- only it wasn't as well known as any of them.

It sounded as promising as iTix or Bits2Go or NexTag or HotPaper or GetMedia or any of the dozen other companies with which I'd interviewed. And it was better than Yahoo, which after weeks of hectoring, had finally relented and offered me a marketing job that would have required me to sell my children to survive. I was burning up with Valley fever and the only cure was to join a cool startup. So when Google agreed to interview me, I threw some resumes into a briefcase, got in my station wagon and drove north to Mountain View for a little chat.

Uh, Karen, can you clean this up a bit?

Jeez, this blog is getting ugly. I've always wanted to try AdSense so I opened an account and used the Blogger interface to do it. I thought the ads would show up in the right hand column, tastefully tucked in below the "I power Blogger" button. But above the posts? That's not helping this page's architectural purity.

Whenever I had design issues at Google, I'd call on our webmaster, Karen. She started a week before I did and we worked together so long we came to believe that we shared a brain. Entirely self-trained in HTML and design, she turned this into this, hired and trained logo god Dennis Hwang, acted as security chief at the first Google Dance and had more influence into the overall site design than just about anyone else on the staff. I don't think I've ever worked with a more dedicated or productive person than Karen. Always good-natured, meticulous and relentlessly true to user-friendly design, Karen was the heart and soul of what made something "look Googley."

And boy, did she ever keep me on track.

"Doug, where's that final copy?"
"Doug, this is ready to ship. Have you signed off on it yet?"
"Doug, we launch tomorrow. Am I gonna get the text before midnight?"
"I'm waiting..."

It's a good thing she was there, or I likely would still be polishing the text for the launch of Google Image Search.

Karen... If you're listening, can you, you know, fix this?

I become unstuck in time

"For that feature to be content-ready would take six weeks."
"We're already behind schedule and code ships on Friday."
"We need help with the daily propagation of Delta One."

All I had wanted was a grande coffee frappucino and I suddenly found myself flashing back in time to life in the GooglePlex. I'd stopped by the Starbucks at Hamilton and Bascom and given no thought to the crowd of casually attired young men and women in their Tevas with socks, logoed polo shirts and pink hair.

"I fixed the HTML, but the structure was all wrong."
"We need a week of a full-time developer, so it's not a trivial problem."

They weren't Google staffers, but eBay worker bees swarming out of their nearby headquarters for a mid-afternoon hit of caffeine and California sunshine. They were speaking a different dialect of a foreign language I once knew quite well. I felt compelled to listen and had to restrain myself from jumping in mid-conversation as an old-timer explained to a curious neophyte the difference between process management and product management.

It all came rushing back. Back-to-back meetings with engineers preparing for launches, conversations in line at Charlie's cafe while holding out my plate for Cauliflower Provencal, 2 AM email exchanges with earnest straight-outta-college associate product marketers. And I realized how much had already slipped away. I was losing not only my ability to understand the lingo, but my ability to speak in Google's voice. It was a voice I had worked very hard to develop in descriptions of our services, newsletters, April Fool's jokes, product tours, promotions, recruiting campaigns and company positioning statements.

I didn't exactly miss it all, but I didn't want to forget it either. So I started writing myself notes and typing them up while sitting wide awake at 5AM with absolutely no urgent email to answer. I was able to reconstruct a rough outline of the major projects I'd worked on and the things I'd found interesting about them at the time. It's not all organized in chronological order. It's not full of hot gossip about who was caught peeking down who's toga at the company ski trip or the three secret HTML tags that will absolutely guarantee a PageRank of eight or higher. There are no deep revelations that are likely to move the stock price (full disclosure: I still own some) or cast light on the inner workings of Sergey's soul. It's just an impressionistic look at Google from the perspective of a guy who worked there, with a bias toward how the brand was built.

If you care, read on. If not, be sure to (text deleted by request of Google AdSense) Someone's gotta pay for these pixels.

"Hi. My name is Doug and I haven't hyped Google for 8 months..."

In November 1999, I left a very secure job with Big Media for a startup technology company that I was pretty sure would be bankrupt within six months. Why would a 41-year-old father of three take a $25,000 pay cut to work with a bunch of guys who still got carded when they ordered beer with their pizza? It's a long story, but one I now have lots of time to tell. This blog is partly about that, but mostly about what happened during the following five years and three months, while I served as Director of Consumer Marketing and Brand Management for Google.

For the last eight months, I've been gathering my thoughts in preparation for writing a book. That may still be forthcoming, but the more I think about it, the more convinced I become that a book would not be the Google way to do this. A book would take months to write and produce. It would be static and couldn't be modified quickly. And it would be hard to search, especially if it were copyrighted.

A book would also reflect just my view of things. Of course, I'd like to believe my view is always the correct one, but after twenty years of marriage, I'm pretty sure that's not the case. And if there was anything Google's staff produced more quickly and in greater quantity than search results, it was diametrically-opposed opinions. So I've called this blog "Xooglers" for "ex-Googlers," who I invite to comment on what I post, and to add their own recollections of the company.

Others are welcome to post as well, though I ask that you resist the temptation to go off-topic or add spam-links. I'm pretty sure that if and when this site gets going, it will be of interest to at least a few folks at Google. I'd hate for your Viagra posts to interfere with their ability to read the content they actually care about. Not that your PageRank would be affected or anything like that. Remember, "Google Does No Evil^TM." Though... you could make an argument that doing evil to those who do evil is good. Perhaps we should let Sergey decide...