Failure Stories Volume 4

Dan Wolchonok, Head of New Products and former Head of Data at Reforge shares the time he deleted the entire Reforge applications table during their peak sales period

Hi there, it’s Adam. 🤗 Welcome to my weekly newsletter. I started this newsletter to provide a no-bullshit, guided approach to solving some of the hardest problems for people and companies. That includes Growth, Product, company building and parenting while working. Subscribe and never miss an issue. Listen to my new podcast about fatherhood and company leadership - Startup Dad. Questions? Failure stories? Ask or share them here.

---

Welcome to the fourth installment of Failure Stories. This is a monthly series. Here are the last several:

• Volume 1: Elena Verna and Leah Tharin, Head of Growth at Dropbox and PLG Advisor; former Head of Product at Jua and PLG Advisor

• Volume 2: Tom Willerer, COO of Reforge and former CPO at Opendoor

• Volume 3: Gina Gotthilf, COO and co-founder of Latitud

Failure. We’ve all experienced it but hate to talk about it. And why is that? As leaders, are we supposed to be flawless; impervious to criticism and mistakes? 

Are we worried that if our teammates, peers, bosses or direct reports find out about some huge flop our ours that we will somehow be less successful than we were a week or two ago or that it might hold us back from that future career opportunity?

In Product we prototype because we know that on a great day only 30% of our solutions will actually work. In Growth we experiment for the same reason. Even the smartest and most talented among us get it wrong, have moments of immaturity, and learn some painful lessons.

The party line is that failure is “OK” because it’s about learning. But, when it comes down to it, is it really okay to fail?

Yes. It is. And because it’s so hard to admit failures I’m publishing this recurring series on the stories of some of the most successful leaders and the struggles along their journey to success. My hope is that this normalizes the losses and mistakes that we’ve all experienced but are too afraid to share. The ones that make us deeply uncomfortable and are a big part of what has forged the successful people you follow today. Those whose advice you clip and save for later, whose newsletters you subscribe to, and whose templates you share.

With each person I ask them two questions:

1. What would you consider to be the biggest professional failure of your career – what’s the story behind it and why was it such a big flop?

2. How did that experience shape you; how did you transform it into a learning opportunity?

---

Don’t let the suit fool you. Dan messes up too.

Today I’m sharing a great story from Dan Wolchonok – Head of New Products at Reforge (he’s working on the amazing Artifacts product currently), former Head of Data at Reforge and former Director of Product Analytics at Hubspot. Don’t let the polished, suit-and-tie look fool you; Dan messes up like a total pro. In today’s issue of Failure Stories we’ll talk about the time he deleted Reforge’s entire applications table during their peak selling season.

Dan Wolchonok is a very accomplished and highly intelligent person. He graduated from Tufts and holds an MBA from Yale. He had a 5+ year career at Hubspot, including working on their Growth team with Brian Balfour.

In 2018 he joined Reforge as their Head of Product and Analytics. The company was small (5-10 employees) but revenue was through the roof and they were running two educational cohorts per year. This was the sole way that they generated revenue for the business. They didn’t have a subscription product; you would buy passes directly for the cohort and it was an intense, concentrated period of time twice per year.

There were only two programs at the time—the flagship Growth Series and Retention & Engagement. Reforge was about to launch Advanced Growth Strategy. It was application time for the Fall 2018 cohort and the next revenue period wasn’t going to be until the Spring of 2019; months away.

Dan was trying to delete and merge applications for individuals, consolidate duplicates and clean up the messiness of the database. They needed to make some harmless adjustments to delete one application so Dan dug into SQL and wrote a simple statement.

Warning: the language in this post definitely reflects the panic he felt at the time.

He recalls:

Still makes me real anxious just thinking about it.

I remember doing the manual sql in Postico.

delete from program_applications
-- where id = 37464

I highlighted the whole thing, ran it, and then screamed out FUCK, NOOOOO!!!!!!! When i saw the row count of all rows in the table affected instead of 1 row affected. Stupid fucking commented out line 😡.

The “--” was commented out the individual application ID so instead of deleting that one application he deleted the entire applications table. That’s the table that holds ALL of the applicant information for the cohort. Every. Single. Paying. Customer.

Dan again:

“The worst part is that I knew better. I would not have allowed a junior employee to be doing that.”

How did that experience shape you; how did you transform it into a learning opportunity?

Despite the immediate panic, as Buford mentions above they were able to recover the data with only ~1 hour of downtime. Dan still gets anxious when he thinks about this.

“Ugh, I was so freaked out that it did some cascading deletes from there. I had an out of body experience. I was beside myself.”

Author note: this is a Dad Joke from Dan.

Dan shared a series of takeaways with me that he has carried forward over the last 5 years since then.

Here are the tactical pieces of advice and lessons learned:

1. Always run things like this in transactions so you can roll it back. The nice thing about doing it in a transaction is that you can run one statement, double check it did the thing you want, then commit the transaction. 

2. If you’re going to be doing something like this regularly, build a quick UI for it. There are so many tools out there that make this easy to do.

3. Setup a read replica and run 99.9% of your queries through that so you never accidentally do this.

Now for some more macro lessons:

1. Don’t hide anything. Tell someone about it immediately. [author note: this demonstrates one of my key leadership principles - “bad news doesn’t get better.”]

2. Your co-workers will save your ass so invest in those relationships. I’m sure if Dan didn’t have that relationship with Buford, dropping a major database table wouldn’t have gotten resolved so quickly or satisfactorily.

3. Have empathy for others when they mess up, everybody does it. This one resonates and contributes to a culture of moving fast and (occasionally) breaking things. If you want people to achieve great things you have to understand that they might get it wrong sometimes. It’s software and we can fix almost anything. Which brings us to our final few points…

4. Always, always, always have backups and be prepared to think through how to fix stuff. Having a deep knowledge of the underlying systems will help you be valuable in fixing the 💩.

Conclusions and takeaways

When Dan told me this story it reminded me of when I was learning how to belay as a rock climber. The instructors make you go through a series of safety checks on your harness, your rope knots, your belay devices, and your communication with your climbing partner. 

I asked them once if they observe a lot of new members messing this up. They said no. Most of the accidents that happen in the gym are caused by experienced climbers who don’t do the safety checks anymore because they don’t think they need to. The same is true with car accidents. Most happen within a few blocks of your home. This is because people start to pattern match through repetition and get on autopilot. Then something changes that they weren’t anticipating and… crash.

Just because you’re incredibly experienced doesn’t mean you can’t make mistakes. In fact, it’s probably more likely that you will because you’re trusted to do more with fewer guardrails. How you recover, the lessons learned, and how you educate others to avoid your mistakes is all that matters.

And also… watch out for that commented code.

If YOU have a failure story that you’d like to share or any feedback on this series please reply and let me know.