Category: Work Life Balance

On Call Practices

May 1st, 2022

One of the downsides to web hosted software is the fact that it’s always supposed to be available. Unfortunately, outages happen – maybe it’s because of a bug in the code, an infrastructure outage, or because an army of squirrels chewed through the power lines that feed into us-east-1.

For a lot of companies, that means having someone on-call to take care of issues quickly. How they handle on-call varies wildly from company to company.

What hours is someone on-call? Do you have one unlucky person on-call 24×7? Do you have people who work in multiple regions around the world and use a Follow The Sun pattern (and if you do have this setup, is the on call only active during the weekend and just deal with issues during the week as they arise)? Something else?

How long is someone on-call? Are they just on call once every X weekends because you’re able to Follow The Sun? Are they on call for an entire week? How many people are in your rotation?

What triggers an event? Are the triggers different during business hours vs after hours? Are the alerts so broad that an endpoint returning a 400-series error causing a page to go out? Do you trigger on disc space or processor load warnings? Do you trigger on the system going down?

What is the SLA expectation on response time? Do the SLA expectations change based on whether it’s during or outside of business hours?

Do the people on-call do production work during their time on call?

If on-call gets activated after hours, do you give the person time off to compensate them?

How often does on call get activated?

What happens after an on call issues is resolved?

What SHOULD (Probably) Be Happening

If you are able to, you should probably Follow The Sun. This means that you can have on call coverage without someone being woken up in the middle of the night and trying to figure out what the heck is going on at 2am. Unfortunately, this isn’t always possible – especially in smaller companies.

At worst, your on call rotation shouldn’t last more than a week.

What triggers an event should probably vary based on whether your on-call person is currently working during business hours or if it’s an after hours page. Actual production emergencies should always page. After hours, the only things that should page are emergencies and possibly suspicious activity pages.

If you’re running cloud native apps, a lot of your issues should be taken care of via automated processes, so you’re likely going to be down to emergency issues, alerts for suspicious traffic patterns, etc.

If you’re running things on actual physical infrastructure (yes, those still exist – especially inside large orgs with legacy tech or when dealing with sensitive data), things get a little trickier. You’re also going to have to deal with space/processor usage and all of the headaches that come with managing hardware.

I was once on a team that sent an alert email during hours if any of the web apps that they ran threw a 400 series error in the logs. Don’t do this. For the love of your team’s sanity, don’t do this. There are a lot of reasons a 4xx return code might be sent back and most of them aren’t a problem with your system (this is especially true if you’re running APIs that are exposed to consumers other than yourself).

Your SLA on response time with your team should also probably vary based on the severity of the issue and when it happens. Part of this is going to depend on how it impacts your business/org. If it’s after hours and an outage isn’t really going to cost you much in terms of either money or reputation, is it really going to matter if it’s 20 minutes before your person is able to log in and resolve the issue?

During the day (provided that is when most of your business load happens), yes the expected turnaround time should be pretty quick. Don’t expect your on call person to respond within 5 minutes on the weekend unless lives are at stake though (and, yes, I’ve been on a team where they expected you to respond within 5 minutes. Everyone hated being on-call on that team).

If you have a dedicated person each week during business hours handle on call pages, that person should NOT be doing production work. Don’t have them working on upcoming features. Instead, leverage the fact that they’re going to be in the troubleshooting mindset by having that person focus on finding and improving things that already exist in the system.

In the long run, this will result in a more stable environment that (hopefully) will lead to even fewer alerts. That’s a good thing.

If you page someone after hours, make it up to them. If you page them at 3am, you’ve ruined their sleep. If you page them on the weekend, you’ve ruined time they need in which to live their lives. At the very least, give them a comp day in return.

It may not make up for the night’s sleep, but it will help your people recover. It will also serve as a disincentive for you to page people and, instead, work on fixing the actual root cause. I’ve seen too many companies (many of them large orgs) who just shrugged as their systems never improved and their people were always exhausted because it didn’t cost them anything in their view (who counts people leaving in those calculations, after all?).

After hours on call pages should be rare. Ideally it should never happen, but if it happens more than once or twice a quarter (at MOST), you have some serious problems. Either you’re paging people for every “error” in a system or your system is so unstable that you shouldn’t be doing any new feature work until you sort your crap out. Full stop.

Sometimes you have the worst of both worlds. The team I mentioned before would send at least one on-call page almost every weekend (no joke) and roughly 3 in 4 pages were for non-issues or things that should only be a warning alert in the system. Real emergencies also occurred too often, but the root causes were never fixed “because reasons”. People literally got anxiety on the week leading up to their being on-call.

After an on-call issue is dealt with, you need to resolve the root cause of the issue. As a general rule, the person taking care of the issue at the time is only expected to do enough to get the system back into a functional state. The underlying problem is still going to be there.

Do a root causes analysis of the issue during normal business hours. The emergency is over and this is now an operational issue. If there’s an issue that you can pinpoint, fix it. No excuses unless this will cause your business to go under because you’re on the ragged edge funding wise or you’re literally decommissioning the service in question tomorrow.

If you can’t find the root cause of the issue (which happens sometimes), are there things that you can monitor which will alert you to the likelihood that it’s going to happen again? The error may not occur exactly the same way again, but even intermittent issues generally follow a pattern if you can figure out what it is.

Nobody likes being on call, but with some planning and work, it doesn’t have to be terribly stressful to be the one carrying the pager for the week. Doing things well will not only help make your systems more stable, but will decrease the chance that your people walk out the door because they’re too exhausted and burned out to function.
Agile Is About Sustainable Pace

April 17th, 2022

You often hear people talking about how “fast” agile lets you work. It’s something touted by a lot of “Agile Consultants” and is eaten up by people in upper management.

I even had one team lead on a project I was on brag about “how much you get done” and literally how exhausted you were at the end of the day as though that was a good thing. He also said that they used pairing as a way of “keeping you honest” because you couldn’t stop during the day. Sadly, I’m not exaggerating. Between that and other behaviors, he’s made the List of people I will not willingly work with again (it’s a really short List considering how long I’ve been doing this).

If this is the mindset you have about agile – especially if you are in a position of authority in your company, it’s time for a little reeducation.

Agile isn’t about “going fast.” It’s about going at a sustainable pace over the long term. Yes, I realize that terms like “sprint” indicate that you’ll be running at top speed all of the time, but those were incredibly poorly chosen marketing words.

Software development (indeed the creation of any reasonably complex system) is more like a marathon. It’s not something that can be done in short order, and we need to view it through that lens. If you try to go at top speed for an extended period of time, the only thing you will accomplish is wearing yourself out without having achieved your objective.

And that’s the best-case scenario. In the worst case, you will injure yourself to the point that you’ll be unable to do anything else at all.

For an organization, the consequences of trying to work at a fast, unsustainable pace are usually a combination of having the things you do ship being full of defects, people not being motivated to do anything at all after a while, people taking a lot of sick days just to be able to rest, and eventually people leaving your company in large numbers.

With all of that institutional knowledge walking out the door, you’re not going to be able to maintain the things that you did manage to ship. To add a further bonus, you won’t be able to hire the people that you want because none of them will be willing to work with you. The whisper network is real and we all talk to each other.

For an individual, the consequences can be even more severe (as is usually the case). They include burnout (at best), a now permanent medical condition like hypertension, diabetes, etc or (in the worst cases) ending up in the hospital because you’ve had a heart attack.

I’m not exaggerating about any of these. I’ve seen otherwise healthy people (self included) suffer long term medical issues from overwork. The heart attack scenario isn’t an exaggeration either – I can think of at least one coworker who suffered that fate. To make matters worse, one of their executives visited him in the hospital and had the gall to ask why he wasn’t studying software related material instead of following his doctor’s orders to do nothing but relax.

I know it sounds like a sick joke, and I wish it were, but this is the reality of things that I’ve seen. I hope it goes without saying that the exec in question is also on my List.

I told you it was a high bar to become a member of that club. Most of the people on it are there because they abused the people around them.

All of that said, I hope we can agree that none of these outcomes are things we actually want either as companies trying to get work done or people trying to live a meaningful life. So what’s the answer to help prevent all of this?

Stop thinking about going full speed all of the time and work at a sustainable pace instead.

The first step to doing this is to realize that the definition of “sustainable” can change day to day – someone may be on vacation, someone may have a bad day or feel out of sorts, your team may be encountering outside problems (AWS going down or a package disappearing from NPM would be relatable examples lately), or the people on your team may be absolutely killing it that day and getting an amazing amount done. All of these things will affect what is “sustainable” for that day.

Think of the long-term instead of day-to-day or “sprint” level. Stop obsessing over having a “stable velocity” (or, even worse, obsessing about it always going up). As long as there aren’t wild swings in your velocity, you’re probably fine.

Stop encouraging (or worse, enforcing) overtime. Realize that working extra hours is a signal that something is broken. Sure, it happens sometimes, but those times should be incredibly rare and you should fix what caused it afterward.

If you do end up having your people work overtime because of an emergency, make sure they get time off to compensate them for that to allow them to rest. In fact, if it was more than a couple of hours, I’d advocate giving them more time off than they spent on OT in order to recover. The reason for this is because it always takes you longer to recover from something like a lack of sleep or extreme stress than it took to get you into that situation.

Encourage your people to spend time thinking about the problem they’re solving instead of obsessing that their hands always be on the keyboard. Obsessing about how much a software developer is typing is about as useful as obsessing about how much a doctor is actively cutting open a patient. The same goes for encouraging your people to spend some time every day/week to do professional development during work hours (and for the same reason).

Make sure your people actually take time off when they need it. Make sure they take a vacation every year. Make sure they stay home and rest when they’re sick. Don’t make them feel like they are going to earn points for working themselves into the ground. And if they are taking time off, anyone who contacts them about work better have an apocalyptically important reason for doing so.

My favorite boss as of writing this used to make me take a 3 day weekend every month because of the level of work I was doing on the project (to say that I was punching above the weight class I was hired for was an understatement and he realized that). Personally, I have sent people home on my team who came into work sick despite their protests that they could still work.

All of these things let your people work at a pace that can be sustained. They leave work feeling like they got something done. They take time off when they need it. They come into the office feeling refreshed. They may even look forward to starting work in the morning (what a concept, right?).

Maybe they’ll even tell the people they know about how amazing it is to work with you as a company and you’ll have your pick of potential employees. It’s one heck of a way to build a pipeline, and it’s even cheaper in the long run than running your people into the ground and replacing them when they quit.

Treat people well and fairly, make sure they get the rest they need, and watch your teams get work done that is not only amazing but is more likely to solve your actual business problems.

Most of us in this field want to do a good job (the same is true of most people in whatever profession). It’s the things that employers and managers do to “make us go faster” or “keep us honest” that end up slowing us down or make us leave.
Taking Time Off Is Normal – You Don’t Have To Explain Yourself

March 27th, 2022

As a culture, the United States has a real problem with the obsession that people always be working. People put off or neglect their vacation time to be “more dedicated”, they work even when they’re sick and should be resting, they’re shamed for not working overtime if they’re salaried, and some employers don’t want to talk to candidates who aren’t currently working somewhere else.

This is unhealthy in the best of times, but going into year three of a pandemic, it’s downright cruel.

Life is meant to be lived and sometimes that means taking an extended period of time when you just let yourself exist. Go to the lake, do a bunch of reading, ride a pogo stick across Michigan, learn to literally juggle chainsaws…

Okay, maybe not that last thing, but you get the idea.

Record numbers of people have left jobs in the last two years and taken time for themselves. Some of them took time for a sabbatical, some took time to introspect and figure out life in the circumstances we find ourselves in, and some of them just took time to decompress and relax. All of these things are not only perfectly fine, they’re laudable.

The landscape is starting to change, and in some ways that’s a positive thing. We may very well be living through one of the largest labor movements in living memory (and the recent reports of service industry jobs successfully unionizing strongly suggests that we are).

The United States has been far, far behind a lot of the rest of the world in terms of quality of life for people in the working class (I include white collar people in this. If you aren’t able to live off of just investments, you’re working class. Deal with that realization however you need to). Thankfully, it seems like we might be seeing the signs that we’re starting to catch up.

Some recruiters and hiring managers, however, disagree and are insisting that candidates “explain themselves” with regard to what they were doing with the time that they weren’t actively working for a business. I even had a recruiter go so far as to say that I needed to add entries to my resume to explain away the time I took for my sabbatical and that I needed to think about how I’d explain it to companies.

I’m not the only person I know who has experienced this either. It’s been a thing for a long time; most certainly longer than the pandemic we’re in the middle of.

However, it’s a really bad move by recruiters and employers alike for a few reasons past the legal considerations.

First, it shows that you consider your employees (or potential employees) to be property rather than to be people because you are showing that your belief is that a person only has value as long as they are producing something for someone.

I’ve said it before and have no problem repeating myself. You do NOT own the people that work for you. You are only in charge as long as the people who work for you decide you are. The moment they decide to leave, any illusion of power and control that you had vanishes.

Second, the market is heavily in favor of the employee right now, especially in tech. If you display this sort of attitude, you are probably going to go to the back of the line with regard to places the very people you want to attract are going to consider.

In other words, if you want to be “the boss” eventually you won’t have anyone to be “the boss” of because the talent you want most will refuse to work with you and will tell all of the people they know about you. The whisper network is real.

Third, sabbaticals have a very long tradition in a lot of fields. This tends to be more true outside of the United States with three exceptions that come to mind fairly quickly – academia, executive positions, and… technology (this is more true as you get to higher levels – staff, principal, etc).

It’s actually normal for people in those areas to take time off on occasion (usually every few years) to introspect on what they’ve done, do some self-directed research/study, possibly do some speaking or writing, and take some time to just breathe for a bit while considering what they want to do next.

As employers, you should celebrate people being able to do this. It takes a lot of planning and work to pull off this sort of thing – especially if you’re funding it yourself. Yes, by all means, talk about what they did during that time if the candidate wants to (not everything is open for public consumption), but do it with the mindset of mutual interest instead of an interrogation where the candidate has to “prove” something to you.

As for the recruiter who insisted that I “explain myself” regarding my sabbatical, I’ve spoken to more than 50 companies over the last few months as I look for the next place I want to be. I can count the number of hiring managers that made a stink about it on one hand and I can say that wasn’t the only red flag they presented.

All told, most companies and hiring managers thought it was great that I was able to arrange my finances and time to take my sabbatical. Some of them expressed interest in doing something similar. A couple of them talked about how some of their own employees took one and then came back.

There were actually a lot of conversations about the things that I did and learned during this period that were rooted in curiosity and a desire to get to know me rather than from a standpoint that I prove myself worthy of having taken time away from formal work.

I’ll admit that, in my case, part of that may be because I seem to have gotten a reputation in my part of the professional community. I’m not saying that to brag because the thought honestly kind of boggles my mind since, in a lot of ways, I still see myself as … just me rather than someone who leads (sometimes global) initiatives and builds and mentors teams.

My fervent hope is that other people have the same opportunity to grow and live as people instead of just being a cog in a machine whose only perceived value is the work that they are able to do. The only way that we are going to get to that point is for us to normalize that life is for living and that starts with people in positions of authority realizing and accepting that people should be able to take time for themselves.

Part of my promise to you is that, as a hiring manager and interviewer, if you’re across the table from me, you won’t have to “explain yourself” if you took some time off. I may ask if you did anything cool that you’d like to share, but that’s just part of the conversation in order to get to know you.

After all, I hire people, not cogs. I don’t own the people on my team and I don’t want to.

Category: Work Life Balance

On Call Practices

Agile Is About Sustainable Pace

Taking Time Off Is Normal – You Don’t Have To Explain Yourself