The Misanthropic Developer

On Call Practices

May 1st, 2022

One of the downsides to web hosted software is the fact that it’s always supposed to be available. Unfortunately, outages happen – maybe it’s because of a bug in the code, an infrastructure outage, or because an army of squirrels chewed through the power lines that feed into us-east-1.

For a lot of companies, that means having someone on-call to take care of issues quickly. How they handle on-call varies wildly from company to company.

What hours is someone on-call? Do you have one unlucky person on-call 24×7? Do you have people who work in multiple regions around the world and use a Follow The Sun pattern (and if you do have this setup, is the on call only active during the weekend and just deal with issues during the week as they arise)? Something else?

How long is someone on-call? Are they just on call once every X weekends because you’re able to Follow The Sun? Are they on call for an entire week? How many people are in your rotation?

What triggers an event? Are the triggers different during business hours vs after hours? Are the alerts so broad that an endpoint returning a 400-series error causing a page to go out? Do you trigger on disc space or processor load warnings? Do you trigger on the system going down?

What is the SLA expectation on response time? Do the SLA expectations change based on whether it’s during or outside of business hours?

Do the people on-call do production work during their time on call?

If on-call gets activated after hours, do you give the person time off to compensate them?

How often does on call get activated?

What happens after an on call issues is resolved?

What SHOULD (Probably) Be Happening

If you are able to, you should probably Follow The Sun. This means that you can have on call coverage without someone being woken up in the middle of the night and trying to figure out what the heck is going on at 2am. Unfortunately, this isn’t always possible – especially in smaller companies.

At worst, your on call rotation shouldn’t last more than a week.

What triggers an event should probably vary based on whether your on-call person is currently working during business hours or if it’s an after hours page. Actual production emergencies should always page. After hours, the only things that should page are emergencies and possibly suspicious activity pages.

If you’re running cloud native apps, a lot of your issues should be taken care of via automated processes, so you’re likely going to be down to emergency issues, alerts for suspicious traffic patterns, etc.

If you’re running things on actual physical infrastructure (yes, those still exist – especially inside large orgs with legacy tech or when dealing with sensitive data), things get a little trickier. You’re also going to have to deal with space/processor usage and all of the headaches that come with managing hardware.

I was once on a team that sent an alert email during hours if any of the web apps that they ran threw a 400 series error in the logs. Don’t do this. For the love of your team’s sanity, don’t do this. There are a lot of reasons a 4xx return code might be sent back and most of them aren’t a problem with your system (this is especially true if you’re running APIs that are exposed to consumers other than yourself).

Your SLA on response time with your team should also probably vary based on the severity of the issue and when it happens. Part of this is going to depend on how it impacts your business/org. If it’s after hours and an outage isn’t really going to cost you much in terms of either money or reputation, is it really going to matter if it’s 20 minutes before your person is able to log in and resolve the issue?

During the day (provided that is when most of your business load happens), yes the expected turnaround time should be pretty quick. Don’t expect your on call person to respond within 5 minutes on the weekend unless lives are at stake though (and, yes, I’ve been on a team where they expected you to respond within 5 minutes. Everyone hated being on-call on that team).

If you have a dedicated person each week during business hours handle on call pages, that person should NOT be doing production work. Don’t have them working on upcoming features. Instead, leverage the fact that they’re going to be in the troubleshooting mindset by having that person focus on finding and improving things that already exist in the system.

In the long run, this will result in a more stable environment that (hopefully) will lead to even fewer alerts. That’s a good thing.

If you page someone after hours, make it up to them. If you page them at 3am, you’ve ruined their sleep. If you page them on the weekend, you’ve ruined time they need in which to live their lives. At the very least, give them a comp day in return.

It may not make up for the night’s sleep, but it will help your people recover. It will also serve as a disincentive for you to page people and, instead, work on fixing the actual root cause. I’ve seen too many companies (many of them large orgs) who just shrugged as their systems never improved and their people were always exhausted because it didn’t cost them anything in their view (who counts people leaving in those calculations, after all?).

After hours on call pages should be rare. Ideally it should never happen, but if it happens more than once or twice a quarter (at MOST), you have some serious problems. Either you’re paging people for every “error” in a system or your system is so unstable that you shouldn’t be doing any new feature work until you sort your crap out. Full stop.

Sometimes you have the worst of both worlds. The team I mentioned before would send at least one on-call page almost every weekend (no joke) and roughly 3 in 4 pages were for non-issues or things that should only be a warning alert in the system. Real emergencies also occurred too often, but the root causes were never fixed “because reasons”. People literally got anxiety on the week leading up to their being on-call.

After an on-call issue is dealt with, you need to resolve the root cause of the issue. As a general rule, the person taking care of the issue at the time is only expected to do enough to get the system back into a functional state. The underlying problem is still going to be there.

Do a root causes analysis of the issue during normal business hours. The emergency is over and this is now an operational issue. If there’s an issue that you can pinpoint, fix it. No excuses unless this will cause your business to go under because you’re on the ragged edge funding wise or you’re literally decommissioning the service in question tomorrow.

If you can’t find the root cause of the issue (which happens sometimes), are there things that you can monitor which will alert you to the likelihood that it’s going to happen again? The error may not occur exactly the same way again, but even intermittent issues generally follow a pattern if you can figure out what it is.

Nobody likes being on call, but with some planning and work, it doesn’t have to be terribly stressful to be the one carrying the pager for the week. Doing things well will not only help make your systems more stable, but will decrease the chance that your people walk out the door because they’re too exhausted and burned out to function.
Failure Isn’t The Goal Of “Failing Fast”

April 24th, 2022

“Fail fast” is a common motto in tech – especially in the startup world. It’s celebrated that you failed fast, and that’s a positive thing – as long as you know why.

Unfortunately, it’s been parroted as a mantra for so long that a lot of people don’t seem to understand the reasoning behind it anymore. I’ve known a number of people in tech who seemed to think that the point of failing fast was that you failed…

I’ve always kind of boggled at that. I realize that, if you repeat something often enough, it has a tendency to lose meaning, but still – in what kind of world does that make sense? “Yay! We failed. Let’s do it again and light some more money on fire!”

The point of Fail Fast isn’t to fail – it’s to realize that you’re doing the wrong thing as early as possible and adjust course. It’s a way of cutting down on the size of your problem space with a minimum of effort and resources (time, money, etc).

The reason a company or person wants to fail fast isn’t to fail. It’s so they can succeed. The faster you realize you’re doing the wrong thing, you can stop doing that thing, figure out why it’s the wrong thing, and decide what a better approach may be. Then you repeat that cycle.

Eventually, you should find something that works. It’s not failure that’s being celebrated – it’s learning.

In a startup (or even an established business if you’re launching a new product), you may throw a whole lot of features at the wall to see what people actually want and are willing to pay for. A lot of those will be dead ends and you shouldn’t keep spending money, time and effort on them when you figure that out.

If you’re trying to solve a technical problem, you may have a number of approaches in mind that might work. Chances are that you’ll try at least a couple of them before you find one that both works and falls within the budget of time/money/effort that make sense.

As an aside, this is part of why “reusable code” is starting to fall out of favor and is being supplanted by “replaceable code.” Sometimes (especially in startups), you just need to rapidly prototype something that will work for now and then replace it with something more performant later when you’ve got the time and funds to do so.

What you don’t want is for some manager or group to keep going with an idea what is obviously not going to work or provide the value that you need just because they’re afraid to say “this isn’t going to work” or because that’s what they think you want.

This is why the emphasis is put on failing fast.

I once worked for a manager who I asked point blank what would happen if the approach didn’t work or the deadline wouldn’t be met – in other words, what kind of contingency plan we had. His response was that he didn’t give people that option because, if he did, they wouldn’t meet his demands.

That’s the wrong damned answer. For so many reasons.

Failure is always an option in business. In fact, with any given implementation, it may be a better option than succeeding with what you’re trying. If, during the course of working on something, you find a better way to achieve the business value you want to provide and it will take the same (or less) time, cost the same (or less) to produce, and be of a possibly higher quality or be more sustainable, you should really think about adjusting course.

Pulling the macho “failure isn’t an option” BS is just you being insecure and trying to look good to your higher ups because you’re “a strong leader.” (This doesn’t actually make you a strong leader and, if it does make you look better to your higher ups, your org has more problems than we have time to unwind here.)

Being open to people telling you “this isn’t working and here’s why” is not only the sensible approach, but is actually a sign of real leadership.

Yes – be open to an approach failing. Don’t make the point of the exercise to fail, though. The point is to figure out what succeeds as quickly and cheaply as possible and that requires having a learning/growth mindset and being able to admit that you’re approaching something in the wrong way.

So go out there and try things. Learn from the things that don’t work, and try to realize that they aren’t going to work sooner rather than later. Fail fast, figure out what went wrong, then get it right.
Agile Is About Sustainable Pace

April 17th, 2022

You often hear people talking about how “fast” agile lets you work. It’s something touted by a lot of “Agile Consultants” and is eaten up by people in upper management.

I even had one team lead on a project I was on brag about “how much you get done” and literally how exhausted you were at the end of the day as though that was a good thing. He also said that they used pairing as a way of “keeping you honest” because you couldn’t stop during the day. Sadly, I’m not exaggerating. Between that and other behaviors, he’s made the List of people I will not willingly work with again (it’s a really short List considering how long I’ve been doing this).

If this is the mindset you have about agile – especially if you are in a position of authority in your company, it’s time for a little reeducation.

Agile isn’t about “going fast.” It’s about going at a sustainable pace over the long term. Yes, I realize that terms like “sprint” indicate that you’ll be running at top speed all of the time, but those were incredibly poorly chosen marketing words.

Software development (indeed the creation of any reasonably complex system) is more like a marathon. It’s not something that can be done in short order, and we need to view it through that lens. If you try to go at top speed for an extended period of time, the only thing you will accomplish is wearing yourself out without having achieved your objective.

And that’s the best-case scenario. In the worst case, you will injure yourself to the point that you’ll be unable to do anything else at all.

For an organization, the consequences of trying to work at a fast, unsustainable pace are usually a combination of having the things you do ship being full of defects, people not being motivated to do anything at all after a while, people taking a lot of sick days just to be able to rest, and eventually people leaving your company in large numbers.

With all of that institutional knowledge walking out the door, you’re not going to be able to maintain the things that you did manage to ship. To add a further bonus, you won’t be able to hire the people that you want because none of them will be willing to work with you. The whisper network is real and we all talk to each other.

For an individual, the consequences can be even more severe (as is usually the case). They include burnout (at best), a now permanent medical condition like hypertension, diabetes, etc or (in the worst cases) ending up in the hospital because you’ve had a heart attack.

I’m not exaggerating about any of these. I’ve seen otherwise healthy people (self included) suffer long term medical issues from overwork. The heart attack scenario isn’t an exaggeration either – I can think of at least one coworker who suffered that fate. To make matters worse, one of their executives visited him in the hospital and had the gall to ask why he wasn’t studying software related material instead of following his doctor’s orders to do nothing but relax.

I know it sounds like a sick joke, and I wish it were, but this is the reality of things that I’ve seen. I hope it goes without saying that the exec in question is also on my List.

I told you it was a high bar to become a member of that club. Most of the people on it are there because they abused the people around them.

All of that said, I hope we can agree that none of these outcomes are things we actually want either as companies trying to get work done or people trying to live a meaningful life. So what’s the answer to help prevent all of this?

Stop thinking about going full speed all of the time and work at a sustainable pace instead.

The first step to doing this is to realize that the definition of “sustainable” can change day to day – someone may be on vacation, someone may have a bad day or feel out of sorts, your team may be encountering outside problems (AWS going down or a package disappearing from NPM would be relatable examples lately), or the people on your team may be absolutely killing it that day and getting an amazing amount done. All of these things will affect what is “sustainable” for that day.

Think of the long-term instead of day-to-day or “sprint” level. Stop obsessing over having a “stable velocity” (or, even worse, obsessing about it always going up). As long as there aren’t wild swings in your velocity, you’re probably fine.

Stop encouraging (or worse, enforcing) overtime. Realize that working extra hours is a signal that something is broken. Sure, it happens sometimes, but those times should be incredibly rare and you should fix what caused it afterward.

If you do end up having your people work overtime because of an emergency, make sure they get time off to compensate them for that to allow them to rest. In fact, if it was more than a couple of hours, I’d advocate giving them more time off than they spent on OT in order to recover. The reason for this is because it always takes you longer to recover from something like a lack of sleep or extreme stress than it took to get you into that situation.

Encourage your people to spend time thinking about the problem they’re solving instead of obsessing that their hands always be on the keyboard. Obsessing about how much a software developer is typing is about as useful as obsessing about how much a doctor is actively cutting open a patient. The same goes for encouraging your people to spend some time every day/week to do professional development during work hours (and for the same reason).

Make sure your people actually take time off when they need it. Make sure they take a vacation every year. Make sure they stay home and rest when they’re sick. Don’t make them feel like they are going to earn points for working themselves into the ground. And if they are taking time off, anyone who contacts them about work better have an apocalyptically important reason for doing so.

My favorite boss as of writing this used to make me take a 3 day weekend every month because of the level of work I was doing on the project (to say that I was punching above the weight class I was hired for was an understatement and he realized that). Personally, I have sent people home on my team who came into work sick despite their protests that they could still work.

All of these things let your people work at a pace that can be sustained. They leave work feeling like they got something done. They take time off when they need it. They come into the office feeling refreshed. They may even look forward to starting work in the morning (what a concept, right?).

Maybe they’ll even tell the people they know about how amazing it is to work with you as a company and you’ll have your pick of potential employees. It’s one heck of a way to build a pipeline, and it’s even cheaper in the long run than running your people into the ground and replacing them when they quit.

Treat people well and fairly, make sure they get the rest they need, and watch your teams get work done that is not only amazing but is more likely to solve your actual business problems.

Most of us in this field want to do a good job (the same is true of most people in whatever profession). It’s the things that employers and managers do to “make us go faster” or “keep us honest” that end up slowing us down or make us leave.
People Are Not “Resources”

April 10th, 2022

Look at a job listing or sit in on an executive meeting and you’re likely to hear them refer to the people at a company or the people that company wants to hire as “resources”.

I’m here to tell you that, if you do this, you need to stop right now. People aren’t “resources” – they’re PEOPLE. They have hopes, fears, dreams, and aspirations just like you do. Stop dehumanizing them, because you’re not only doing harm to the people you refer to that way, you’re doing harm to your organization and even to yourself.

Before you start your protest of “I’m not dehumanizing anyone,” yes you are and you need to realize that you’re doing it. That’s the first step to stop doing it.

A resource is something that you use until it breaks and then you throw it away and go get another one off the shelf. “Fungible resources” are even worse – that’s saying that Thing A is a 1 for 1 fit for Thing B.

People aren’t identical. They never will be. Even if you look at them purely from a skills perspective, no two people are going to be alike (I know so many HR people, managers, and executives that have fought me on this. Here’s a hint – they’re all wrong. We’ll get to why later).

If Priya or Christina or Bob quit, you can’t just go to the employee store, pick Kyle off the shelf, bring him to the office and tell your team “This is Kyle. He’s your new Bob. I expect nothing about the way you work to change.” It doesn’t work like that because Kyle and Bob are not interchangeable (this is to say nothing of the fact that it’s going to take Kyle time to learn your environment, needs, etc).

If this is the case, why is it so rampant in the working world? The short answer is that it’s literally done in order to dehumanize the people that work for you. If you only think of people as interchangeable parts and numbers on a spreadsheet instead of as individuals with families, you don’t bat an eye at laying off entire divisions to make your quarterly numbers look good.

Seriously. That’s the whole point. Business schools teach people to use this kind of language (as was pointed out by a friend of mine in conversation the other day). They may not teach people why they use that specific language, but that’s the reason (and it’s been going on long enough that some of the people teaching it may not even realize the reasoning). Since most CEOs come up either through business or legal, the language taught in those arenas dominates the speech used.

We could get into the discussion that it likely started during the Industrial Revolution in the west when hand-crafted parts were replaced with duplicate, mass produced parts, but that’s a whole other conversation.

So, we’ve established that “resources” is the language of the CEO and that it comes from their formal education. Why, then, does it seem like everyone in so many companies use the same language?

Because they want to be seen as part of the in-group. It’s a way for the people below the CEO to say “I’m just like you. You want to keep me around and promote me.” This then filters down through every layer of the organization for the same reasons it happens at the top – it’s an attempt to be part of the group.

The problem is that it’s actively harmful in several directions.

The language we use shapes the way in which we think. If we use language that dehumanizes people, we start to think of them in a dehumanized way – as something that is easily replaceable which we shouldn’t have to consider for anything other than their immediate use to us.

We literally have a word for that – sociopathy. And, no, that’s not an exaggeration. Think of the behavior of your average medium to large business (and the people who run them) and then look at the list of traits for Antisocial Personality Disorder. You’ll see one heck of an overlap.

That is not a healthy situation to put people in, and we not only do this daily in business, we try to frame it as some weird kind of virtue. That in itself is both broken and alarming.

By using dehumanizing language, you are literally dehumanizing the people you use it on. We’ve already stated that. What we haven’t said is that, by making it okay to dehumanize the people below us, we signal to the people above us in the org chart that it’s okay to dehumanize us.

It’s starting to hit a little differently now that it impacts you, isn’t it?

Furthermore, all of those DEI efforts that companies like to brag about suddenly look very empty. You can’t have Diversity, Empowerment, and Inclusion if you don’t respect people as people. You don’t care about the diversity of a paperclip. In fact, if the paperclip is “too different” it might not be “useful” given this mindset. You most certainly don’t empower a paperclip or include it in your discussions on how to run your business.

This is why so many DEI initiatives inside large orgs fall flat. They want to be able to say they value people without actually doing so or changing anything about how they behave in the larger scheme of things.

People aren’t paperclips. The differences between one person and another can be an incredible advantage to a business because they allow us to look at problems from different angles in order to come up with answers that might not be obvious to any single person in the group. It’s one of the reasons that, to a business, real DEI is actually an amazing tool for growth (apart from, you know, treating people like people which we should be doing anyway).

We live in a global economy. The customers for a business are almost certainly going to be much more diverse than the largely white, male leadership of your org. If you want to solve problems that appeal to a wide variety of demographics, you can’t do it by treating your people as replaceable cogs. Trying to do so is going to actively hurt your business in the long run.

I’m not framing it this way to do my own form of dehumanization on people by reducing them to an argument that features business utility and increased profits or viability. I’m doing it so that people in charge of a business can look at this from a sense of enlightened self-interest instead of just going “yeah. Sure. Moral argument, but morals don’t pay the bills.”

People who know me well can tell you that I’m a big fan of the term “enlightened self-interest” when it comes to framing arguments. It tends to be incredibly effective once you start to get people to see it that way.

Sometimes you have to speak someone’s language in order to get them to understand why they need to change. It takes a lot of energy to redirect the behavior of a person or business – especially when you’re trying to change the prevailing thought and language of a couple of centuries of practices.

So, what can we do to change this trend? The answer to that depends on your position, risk appetite, relative power/privilege, and a variety of other factors.

I don’t expect an entry level employee or someone without a lot of institutional power to have the same tactics and strategy that I do. I’ve literally stopped meetings with high level managers who used the term resources to ask “By ‘resources’ do you mean people? Then say people.”

I’ve even had some success by doing this, but I am coming from a position of influence and I realize it (I’ve been in tech for 20 years and have been in leadership positions for a decent amount of that time). You may not be. There are other ways that you can help change culture. Do what you feel is both right and relatively safe from the standpoint of being able to keep a roof over your head.

If you are in a position to alter the language, however, please do so. If you’re the one writing job listings, remove the word “resource”. If you’re a manager, stop calling your people that word and push back against your bosses and peers when they use that word (point them to this post if you want to – or to the work of countless other people on the subject).

The only way culture is going to change is if we work to change it. Otherwise, the people who set the tone at the top are just going to keep treating us all like numbers on a spreadsheet because, to a lot of them, that’s exactly what we are, and that is something that needs to change for so many reasons.
Not All Growth Has To Be Uncomfortable

April 3rd, 2022

During my career, I’ve constantly heard about how learning new things has to be uncomfortable, how you have to just throw yourself into the deep end, etc. People will also try to equate learning things with how you have to “exercise hard” in order to build muscle and get stronger (which is another thing that isn’t true depending on the kind of strength you want to build). The obsession and expectation is that basically any kind of growth has to hurt.

I’m here to tell you that’s not only untrue, it’s actively harmful to make this the expectation.

Why is it harmful? Setting the expectation that learning has to be uncomfortable means that you are, at a minimum, going to cause people with various levels of anxiety to self-select out of this career. Speaking as someone who has anxiety (a fact which may surprise some people who know me), the anticipation that something is going to be difficult or painful can make it daunting, if not impossible, to start something.

This expectation that learning is going to be uncomfortable will also work against people with depression, PTSD, C-PTSD, etc for similar reasons. These groups contain a lot more people than you’d probably suspect, and that makes “stress is normal during learning” a very dangerous and harmful gatekeeping device.

So how do we fix it? We start to normalize, from a social standpoint, the reality that a lot of learning can be done gradually and in a low-stakes environment.

Yes, we’ve all been in situations where we’ve had to figure out the answer to a problem when production blew up and the stress was at a high level. Some of us have also had the dubious pleasure of learning tons of new technology on the fly because the requirements of our jobs changed rapidly. And I’ll be the first person to admit that you can learn a lot like that in a short period of time, but I’ll also be the first person to admit that learning like that is extremely scattershot and stressful.

Not to mention that most of the time, the things we learn come in as a trickle, not as a flood. They happen as the result of solving relatively mundane problems as we work or poking at a project idly on some random evening. In fact, they often happen so gradually that we don’t even realize they have happened until we look back and see how far we’ve come.

The best way I’ve found so far to take the stress and pressure off of learning new things is to try to foster a sense of kaizen (roughly translated – “continuous improvement”) in both myself and on my teams where learning is concerned. If it becomes part of your regular routine, it tends to become a lot less stressful.

Make it part of your regular workflow. Explicitly make it known that not all of your working hours are going to be spent hands-on-keyboard shipping features, but rather that some part of the day/week is going to be used to learn. If you don’t make it an explicit expectation on your teams, people will often sacrifice their time for learning in order to “catch up” or “make a good impression” and this will cause them not only to stagnate from a skills perspective, but to become even more insecure because they feel like there “aren’t enough hours in the day”.

I’ve seen a number of places where they said that they wanted people to continue learning, but their actions showed that they only valued being a feature factory. This is not a good place to be in. You have to not only say that you want your people to learn, you have to set the tone in your organization so that it’s socially normalized and acceptable to be able to answer “what are you doing?” with “Learning X”.

From an actual learning standpoint, I’ve found that one of the most effective ways to learn new things in a low stress way is to take it in stages. I’ll decide that I want to learn Thing X and then break it up into manageable chunks – “manageable” is the key word here. In fact, I’ll often try to figure out what the simplest thing that I can do next to learn the subject is. Being overly ambitious is a recipe for stress when things don’t work, and we’re trying to avoid that.

Let’s use the example of learning a new language.

First, I’ll want to learn the basic syntax, so I’ll probably watch a couple of videos or read a quick start tutorial.

After that, I’ll write a little code. Nothing too complex – hello world, a couple of functions to do simple math and work with strings, etc. Essentially just enough to figure out how to write “working” code, simple functions, and possibly basic output.

Next, I’ll learn to write a test in it. This may seem like a big jump (and if you aren’t well versed in TDD it may be – these are my steps. Yours may be different). Being able to write a test means that I can iterate on learning new things faster because I can write code snippets to learn different concepts without having to write an entire program around each of them.

After I’ve figured out the basics of a language and how to test, we’re on to the concept of Breakable Toys outlined in the book Apprenticeship Patterns. Breakable Toys are programs that literally don’t matter in the scheme of things. It doesn’t matter if you screw them up because nothing is depending on them, so don’t worry about making mistakes.

For these, I may use some code kata (which are available online) or start to work through exercises in the book Exercises For Programmers (which basically starts at “hello world” and works up to consuming and creating APIs).

I’d recommend putting your Breakable Toy into source control and commit frequently (make it a private repo if you’re nervous about people seeing your mistakes as you learn). This will allow you to get back to a known good state if you make a mistake while you’re working. I find that it’s a lot less intimidating to try things if I know that I can roll back broken changes.

After I get to this point, I’ll probably spend some time learning the finer points of the language – being able to program idiomatically in the language for example. I’ll also probably spend time learning about security issues and how to resolve/avoid them (this is something that’s in my wheelhouse as a general skill so I try to keep up). These are stretch goals for me, so don’t think that you have to get to this point to say that you can work in a language.

At this point, you’re literally creating functioning applications and have done so in a way where the consequences of making a mistake are essentially the frustration of figuring out what went wrong. It may be a slightly longer timeline to learn the language, but the stakes are much much lower. Congratulations, you have a new, useful tool in your toolkit. (Now, put it on your resume and in your notes for your end of year review)

The added bonus is that, by breaking it up like this, you can do it a few hours a week and make noticeable progress. In all honesty, when I follow the workflow I outlined, none of the steps tend to take more than a few hours with the exception of writing more complex programs later in the learning cycle (I wasn’t kidding about doing the next, simplest thing possible).

Also remember to celebrate the wins at every step. You are making forward progress every time your tests turn green. Every time you figure out why the thing you just tried didn’t work, you’ve grown as a developer. Take a moment to realize that and give yourself a pat on the back.

Pain doesn’t have to be a part of most learning. In fact, a lot of times, it’s better when it isn’t a part of the process. We need to accept and normalize this mindset for ourselves and others.
Taking Time Off Is Normal – You Don’t Have To Explain Yourself

March 27th, 2022

As a culture, the United States has a real problem with the obsession that people always be working. People put off or neglect their vacation time to be “more dedicated”, they work even when they’re sick and should be resting, they’re shamed for not working overtime if they’re salaried, and some employers don’t want to talk to candidates who aren’t currently working somewhere else.

This is unhealthy in the best of times, but going into year three of a pandemic, it’s downright cruel.

Life is meant to be lived and sometimes that means taking an extended period of time when you just let yourself exist. Go to the lake, do a bunch of reading, ride a pogo stick across Michigan, learn to literally juggle chainsaws…

Okay, maybe not that last thing, but you get the idea.

Record numbers of people have left jobs in the last two years and taken time for themselves. Some of them took time for a sabbatical, some took time to introspect and figure out life in the circumstances we find ourselves in, and some of them just took time to decompress and relax. All of these things are not only perfectly fine, they’re laudable.

The landscape is starting to change, and in some ways that’s a positive thing. We may very well be living through one of the largest labor movements in living memory (and the recent reports of service industry jobs successfully unionizing strongly suggests that we are).

The United States has been far, far behind a lot of the rest of the world in terms of quality of life for people in the working class (I include white collar people in this. If you aren’t able to live off of just investments, you’re working class. Deal with that realization however you need to). Thankfully, it seems like we might be seeing the signs that we’re starting to catch up.

Some recruiters and hiring managers, however, disagree and are insisting that candidates “explain themselves” with regard to what they were doing with the time that they weren’t actively working for a business. I even had a recruiter go so far as to say that I needed to add entries to my resume to explain away the time I took for my sabbatical and that I needed to think about how I’d explain it to companies.

I’m not the only person I know who has experienced this either. It’s been a thing for a long time; most certainly longer than the pandemic we’re in the middle of.

However, it’s a really bad move by recruiters and employers alike for a few reasons past the legal considerations.

First, it shows that you consider your employees (or potential employees) to be property rather than to be people because you are showing that your belief is that a person only has value as long as they are producing something for someone.

I’ve said it before and have no problem repeating myself. You do NOT own the people that work for you. You are only in charge as long as the people who work for you decide you are. The moment they decide to leave, any illusion of power and control that you had vanishes.

Second, the market is heavily in favor of the employee right now, especially in tech. If you display this sort of attitude, you are probably going to go to the back of the line with regard to places the very people you want to attract are going to consider.

In other words, if you want to be “the boss” eventually you won’t have anyone to be “the boss” of because the talent you want most will refuse to work with you and will tell all of the people they know about you. The whisper network is real.

Third, sabbaticals have a very long tradition in a lot of fields. This tends to be more true outside of the United States with three exceptions that come to mind fairly quickly – academia, executive positions, and… technology (this is more true as you get to higher levels – staff, principal, etc).

It’s actually normal for people in those areas to take time off on occasion (usually every few years) to introspect on what they’ve done, do some self-directed research/study, possibly do some speaking or writing, and take some time to just breathe for a bit while considering what they want to do next.

As employers, you should celebrate people being able to do this. It takes a lot of planning and work to pull off this sort of thing – especially if you’re funding it yourself. Yes, by all means, talk about what they did during that time if the candidate wants to (not everything is open for public consumption), but do it with the mindset of mutual interest instead of an interrogation where the candidate has to “prove” something to you.

As for the recruiter who insisted that I “explain myself” regarding my sabbatical, I’ve spoken to more than 50 companies over the last few months as I look for the next place I want to be. I can count the number of hiring managers that made a stink about it on one hand and I can say that wasn’t the only red flag they presented.

All told, most companies and hiring managers thought it was great that I was able to arrange my finances and time to take my sabbatical. Some of them expressed interest in doing something similar. A couple of them talked about how some of their own employees took one and then came back.

There were actually a lot of conversations about the things that I did and learned during this period that were rooted in curiosity and a desire to get to know me rather than from a standpoint that I prove myself worthy of having taken time away from formal work.

I’ll admit that, in my case, part of that may be because I seem to have gotten a reputation in my part of the professional community. I’m not saying that to brag because the thought honestly kind of boggles my mind since, in a lot of ways, I still see myself as … just me rather than someone who leads (sometimes global) initiatives and builds and mentors teams.

My fervent hope is that other people have the same opportunity to grow and live as people instead of just being a cog in a machine whose only perceived value is the work that they are able to do. The only way that we are going to get to that point is for us to normalize that life is for living and that starts with people in positions of authority realizing and accepting that people should be able to take time for themselves.

Part of my promise to you is that, as a hiring manager and interviewer, if you’re across the table from me, you won’t have to “explain yourself” if you took some time off. I may ask if you did anything cool that you’d like to share, but that’s just part of the conversation in order to get to know you.

After all, I hire people, not cogs. I don’t own the people on my team and I don’t want to.
Businesses, Stop Obsessing Over Short Resume Entries

March 20th, 2022
For most of my career, I’ve heard businesses saying that “short resume entries” are a Red Flag in a candidate. It seems to be a little bit less common of a mindset on the east and west coasts than it is in the Midwest (probably because of the startup heavy markets on the coasts). In the last few years, it’s even become less of an issue because of market pressures (which I consider a good thing).

There are a number of problems with this particular “red flag” mindset. First among them is that everyone has a different definition of “short resume entry”. For some people, a couple of 3 month entries set off their alarm bells. For others, anything less than a year is “short”. I’ve even known some people who I would consider to be amazingly out of touch that consider 2-3 years as “short”.

In addition to the fact that nobody knows what “short” is to somebody else, the person yelling “red flag!” has absolutely no idea why the person left that job (and, honestly, in a lot of cases, it’s none of their business). A so-called short entry on a resume could indicate:
- A person found themselves at a bad company.
- They do a lot of consulting.
- They get significantly better offers on a regular basis.
- The company that hired them experienced financial problems and had to lay people off.
- Any number of other things.
We’ll use a couple of examples from my own resume to illustrate this.

I was at one of my positions less than a year because they were a consulting company and one of their largest clients cut their staffing levels extremely hard due to economic externalities (I could explain the effect of falling commodities prices on an agriculture equipment corporation, but I don’t want to bore you). This was such a concerning character flaw on my part that I’ve been contacted by one of the C-level officers that I worked with there to see if I would be interested in joining his new venture.

At another position, I was on staff for less than a year because I was working on a government contract with a set end date and I knew that going in. I was an FTE for the company I worked for, but we knew it was uncertain whether or not I would be staying.

As the contract was ending, my director-level boss started talking to all of the other divisions in the company to see if there were remote positions that I could join because they were so happy with my work. I’ve since been contacted by the company in question two more times to see if I was interested in joining them again. The first time didn’t work out for literal political reasons. We’re currently waiting to see about the second time.

The hilarious thing is that I’ve had hiring managers give feedback that they liked me but considered me a flight risk based on those two entries on my resume. I know I’m not the only one with similar experiences in that regard.

Let’s call this out for what it really is – fear and insecurity on the part of companies who are looking for basically any reason why a candidate might not be the “perfect” fit. They want to feel like they have all of the power and that the candidates should be perfectly loyal and, in the words of several hiring managers I’ve known, “stay around for a long time.”

I’ve been on both sides of the interview table, so I understand what they think the motivation is for saying that. Hiring is stressful for everyone (though the company has a lot more power than they try to act like they do) and making a bad hire sucks (though it’s not as catastrophic for most companies as they pretend). However, 90%+ of the time, the reason for a short resume entry is because of the company the person was working for, not because of anything “wrong” with the person themselves.

The candidate doesn’t have any control over whether or not your company culture is toxic. They don’t have control over your finances. They don’t control how you pay in relation to other companies or your lack of focus on continuing professional development. That’s all on your company.

Let’s be honest, the majority of the risk taken on in a working relationship is taken by the employee and not the company. The employee is almost always going to be in a worse position if the company ends their relationship than a company will be if an employee gives their notice. This means that, most of the time, if there’s a short entry on someone’s resume that person has a really good reason that it didn’t last long.

You need to realize and accept this. When you do, you can work to mitigate those “flight risk” factors by giving people a reason to stay instead of giving them reasons to leave.

Employers – Accept that, at some point, almost any employee can and will leave if given reason to do so. You don’t own the people that work for you, and too many managers try to act like employees owe their lives and loyalty to the company.

I’d argue against asking candidates why entries on their resume are “short” because it’s often none of your business, can surface things that have legal consequences for you (in the case of them being discriminated against based on a protected class for instance), or may even cause the person trauma by making them relive past incidents. If you absolutely feel that you must ask someone why they have short entries on their resume, use it as a lesson in what you need to avoid doing as a company so you don’t lose your people the same way and not as a reason to disqualify them.

Employees – Don’t put up with toxic environments and realize that it’s okay to act with a sense of enlightened self-interest if a much better opportunity comes along. Life is too short to give loyalty to people who don’t deserve it (especially if they’re using it to exploit you).
You Need To Be Visible

March 13th, 2022
For a lot of us in tech, and even in the wider working world, the prevalent thought seems to be that if we do a good job, we’ll get rewarded and things will just sort of work out. This seems to be especially true for people who come from a lower or middle class background (I include myself in this).

The problem is that this isn’t true.

If people don’t know that you did the work, it might as well not have happened when it comes to raises, promotions, bonuses, etc. In a tight market, it can even be the difference between keeping your job and being laid off.

This is a really difficult lesson to learn, because we’re often told “don’t brag” when we’re growing up. The problem is that telling someone “don’t brag” will hurt them in the long run because it’s usually done indiscriminately whenever a child talks about things they’ve accomplished. I can also tell you that people growing up in upper class households don’t hear that sort of thing nearly as often.

It is, among other things, an example of the toxic Protestant Work Ethic and is used far too often to “keep people in their place.” It’s leveraged by racists, classists, and sexists alike and it very badly needs to be consigned to the trash bin of cultural practices.

It’s not bragging to make sure that people know what you’re accomplishing. It’s actually just another form of marketing and advocacy – you are your own greatest advocate. Nobody else cares about your well-being and advancement as much as you do because it directly impacts you. You can’t just wait for people to notice you. It doesn’t work.

Saying “You need to be visible” isn’t all that helpful though, so I’ll give practical examples of how to do this.

Inside the company
- Updates in standup are great, but have a small audience. You can become known as a subject matter expert on your team, but don’t expect the perception of your abilities and value to extend past your immediate team using this approach.
- Use your 1 on 1s to outline, in more or less real time, your wins. This isn’t the only thing you should discuss in a 1 on 1, but make sure you keep your accomplishments on your boss’ radar. This is true for any skip-level 1 on 1s you have too (your boss may not surface your accomplishments up the chain, so try to self advocate)
- Keep a list of your accomplishments as they happen. Use these during your year end review.
- If you have the opportunity, give internal talks/presentations on the things you’ve done. If you shipped a product, do a presentation on it with the stakeholders. If it was a group project, share the credit.
- If you are a manager or lead, PUT A SPOTLIGHT ON YOUR PEOPLE AND THEIR ACCOMPLISHMENTS. Don’t try to take the credit for their work. Leading a team that does good work makes you look AMAZING to most reasonable management chains. It shows that you can effectively lead, delegate, and inspire. That sort of performance can lead to promotions.
Outside the company
- If you’re doing something that you can post about publicly, do so (twitter, LinkedIn, blog, etc). Even if it’s just broad strokes. You may find it boring because you’re in the middle of it, but I guarantee you that it’s interesting to someone.
- If you can’t talk about the product, can you talk about the technical challenges that you’re facing and overcoming?
- Talk about the leadership issues that you’re helping with. If you’re leading a team, what is your style like? How do you help the people that work under you? How do you coach people to the side and above you?
- If you feel comfortable, give a talk on some of the things you’ve learned. Public speaking can be unnerving, but we want to see the people speaking succeed. Seriously.
This list is by no means exhaustive, but I hope it helps. If you feel like you’re being too forceful, take it a little more slowly. You don’t have to do everything on the list at once (though I do advocate keeping the list of accomplishments. It helps not only with year end reviews, but resume writing and talking points in interviews).

As for the backlash that underrepresented folks face for self-advocacy, if you see someone getting abused in that way, call out the person doing it. If you’re in management and your reports are doing it, you need to either make it a Teachable Moment to correct their behavior or give real thought to managing that person out of your organization. To ignore it is a sign that you agree with it.

If you want real Diversity, Empowerment, and Inclusion in your company and the wider culture of your profession, you need to actively work toward it. Making it safer for people to self-advocate is an important step on that journey. After all, “the arc of the moral universe is long, but it bends toward justice” is only true if we put our damned weight on it.
You (Probably) Need a Designated Jerk

March 6th, 2022

We talk about psychological safety a lot in the workplace now, and for good reason – you’re more likely to do a better job, surface issues faster, and offer solutions to problems if you feel like you’ll be taken seriously and treated fairly.

In a psychologically safe environment, everyone would feel free to offer pushback when decisions at any level don’t make sense. They’d ask if this is the correct move for the strategy that has been decided on. They’d even feel safe enough to bring forward ideas to improve the product and process or possibly even surface things that could lead to new revenue streams.

In short, psychologically safe environments are probably the best way to improve not just a product, but a team and even a company. Unfortunately, in my experience, they’re also fairly rare…

In cases where you don’t have psychological safety outside of your immediate team, that team is probably going to need what I’ve come to refer to as a Designated Jerk. The Designated Jerk isn’t someone who acts like what you typically think of as being a jerk – they’re the one who stands up for the team, asks the uncomfortable questions, and gives (hopefully tactful) pushback to levels above them as well as to the side when necessary. In short, they’re someone who helps shield a team from abuse and bad management so they can have safety within the group even if they don’t have it outside.

Ideally you wouldn’t need one, but if you do, it generally takes one of a couple of forms – technical or managerial.

The technical Designated Jerk is probably going to be one of the most experienced techs on your team. They may be the tech lead (and if you’re lucky they are because that usually gives them at least a little official authority), but sometimes they’re just someone who really knows their stuff and leads by example. These folks are usually the ones that will help push back when architectural decisions handed down from above are going to cause problems.

They may also push hard for feature work to be stopped or reduced until a system is stable so on-call becomes less onerous, the cycle time between committing code and it being deployed is reduced so value can be delivered faster, or any number of other tech related issues. They’ll also probably be keeping a finger on the pulse of the team, watching for signs of burnout and pushing back on those above them when needed.

If you’re really unlucky, this person may also be fighting to keep your team from working ridiculous amounts of overtime or to break the Permanent Crunch mentality.

The managerial Designated Jerk is hopefully going to be a manager, but may also be a BA, scrum master, or similar. The reason I say it will hopefully be a manager is that they will, theoretically, have more authority to push back and get breathing room and safety for their team.

If you have a managerial Designated Jerk, they should be working with the team to surface issues that need to be addressed and to formulate arguments and alternatives in order to keep life for the team sane. With this kind of Designated Jerk, you may not necessarily need a technical Designated Jerk to go along with them, but you’re still going to need someone well versed in making technical arguments to help the managerial Designated Jerk be more effective in bringing about change.

One thing both types of Designated Jerk are going to have in common is that they will, at a bare minimum, be able to form their arguments in the form of business cases to show the people above them in the org chart how this will impact the company either short or long term. They will probably have some understanding of how to roughly calculate Return on Investment, Opportunity Cost, how reputational damage can affect sales or hiring, etc.

In cases where you have both a technical and managerial Designated Jerk who work well together, the results can be honestly amazing. Together, they can not only provide cover and psychological safety to their team, they may be able to extend that bubble to other teams that they interact with frequently.

The downside is that being a Designated Jerk can be absolutely exhausting at times – especially when the pushback they provide is ignored or steamrollered. If you end up being this person, check in with the people around you periodically to make sure you aren’t becoming jaded or burnt out yourself because of what seems like a hopeless battle.

Also check in with those around you to make sure that you aren’t transitioning from Designated Jerk to just plain being a jerk. Part of the risk of being the one who offers pushback or ends up saying “no” a lot is that it sometimes becomes part of your personality and you start to let it become part who you are rather than a tool used to improve things around you.

An old friend of mine once gave me a piece of advice that has stuck with me for many years now – “You can either change where you work, or you can change where you work.”

Be passionate about making the world around you better, but don’t let it consume you. Being a Designated Jerk is only one way to accomplish this and, in a reasonable world, it’s not a tool you should have to use too often. If it needs to be used frequently, that may be a sign that it’s time for you (and your team) to take option 2 from my friend’s advice.
Saying I Don’t Know is FINE

February 26th, 2022

“I can live with doubt and uncertainty and not knowing. I think it’s much more interesting to live not knowing than to have answers which might be wrong.” – Richard Feynman

As a culture (both professionally and at-large), we have a real problem with the expectation that we always have the answer. Not just an answer but the answer – the answer that is both correct and, even more importantly, pleasing to the person who is asking the question.

We see it in interviews when we’re asked questions that we would never encounter in our day-to-day work and the literal answer for when we do encounter them is to do a bit of research. Algorithm questions are a good example of this, though my favorite example is probably when a couple of relatively newly minted developers decided to ask me how certain things in Java were implemented in the JVM (as in literally how they were written).

The largest part of our job as software developers, infosec people, devops, etc is research and thinking about possible solutions, tradeoffs, etc. It’s a normal, and indeed healthy, thing for us to say “I don’t know” and then to go off and find answers. We don’t have all of the answers and shouldn’t be expected to.

Even when we do have an answer, there may be a better answer that can be uncovered with just a little bit of looking. After all, bubble sort works and is pretty simple to implement from scratch, but would you actually use it for a production sized dataset?

Over the years, I’ve noticed a correlation between the refusal of people to accept “I don’t know” (followed by some variation of “I’d have to look into it” where appropriate) with people who are either absolutely terrible at their jobs or a nightmare to work with. Correlation isn’t causation, but this one seems to be a pretty strong correlation.

The worst example is arguably one manager I had who would literally make up answers to things in order to avoid admitting not knowing something. When the “answer” turned out to be wrong (sometimes disastrously) he would try to come up with some reason it wasn’t his fault or blame it on “miscommunication”. Even worse, he tried to get my mentees to do the same – going so far as to tell one of them that he was never, ever, supposed to say “I don’t know” to a person above him on the management chain.

This is after I’d spent a year trying to foster the psychological safety in my people to allow them to be comfortable with admitting they didn’t know something and asking for help if they needed it while looking for an answer. Thankfully my mentee came to me afterward and we avoided most of the damage that could have been done if he’d followed my manager’s demands – both from a standpoint of professional development and by not feeding wildly wrong information to upper management.

Saying “I don’t know” isn’t a bad thing. It’s the first step to finding out the answer. It’s a strength, not a weakness. Pretending to know is the weakness – it prevents you from growing and giving bad information can even lead to horrible consequences later.

Foster the mindset in the people on your team that they are allowed to not know an answer, provided they take steps to find a suitable one if it’s needed. Foster that mindset in yourself too and take it as a major red flag if your managers (or even worse, the organization as a whole) try to force you into providing answers that you don’t yet have without giving you the ability to think and look into a problem.