How are software engineers objectively measured when problems span such large difficulty ranges and there are multiple ways to implement solutions?

71 views

Always been curious how “workload” is assigned and estimated for software engineers.

In: 92

That’s a problem no one has really solved yet.
The way it usually works is that you break down big tasks in more manageable sub-tasks. Then you estimate how long each sub task is going to take, based on how long similiar sub-task took in the past, add up all the sub-tasks, then add some allowance for tasks taking longer than estimated, then add some more, and finally you have a number that you can tell your manager.

If you have a team of people you generally know who works faster or slower and assign them a corresponding amount of work.

With a bit of experience it works reasonably well.

Nobody is objectively measured.

The concept of “objectively” measuring somebody’s contributions in the workplace was created in response to blatant unfairness and inequality. As managers and workers struggled with opacity in pay and increasingly large compensation packages for executives, many people demanded answers and clarity for evaluating performance and progress for raises and promotions.

Sure sometimes the metrics can make a lot of sense, especially in more concrete task-oriented work, but mostly this still serves the interests of capitalists and the ultra wealthy by reinforcing the idea that people shouldn’t be equal and providing a convenient pretense for paying executives exhorbitant amounts because of “performance metrics.”

This isn’t to say that no metrics should be used for gathering feedback and encouraging progress through promotions and raises. It’s just to say that they are mostly made-up and should not be considered complete nor infallible. Push back. Ask questions. Share information among co-workers. Unionize. Question executives and wealthy shareholders’ interests.

I am a software engineer and certified SCRUM master.

The answer is very poorly.
The trick is to spread out the bad estimates so the badness cancels out and on average you get something approaching accurate.

The SCRUM alliance recommends having the engineers each give an estimate of how many “points” a task will take. Then taking an average. So probably you’ll get some rough consensus on how many points a task will take.

How long do you take to do a point? Nobody knows, this will change from team to team, that’s literally the reason we’re calling them points rather than something like days or hours.

So at the end of a certain amount of time, we count the total points of all the tasks done.
This gives a rough estimate of how much stuff the team can do in the period of time (a sprint).

So what if the whole team put really big or really small number on their points? This will mean that they’ll also do a huge/tiny number of points per sprint, so you’ll know how much they’ll do the next time.

What if they’re real good at some stuff, but bad at others and they give them the same points? Well them being wrong in both places will cancel each other out. Ditto with the team members being real good/real bad at their jobs.

Of course literally nobody does it right, but it’s fun to dream about what could be.

It’s a combination of experience and guessing. If you or other developers already did similar tasks or know exactly how to implement a task you might get a good estimate. If you’re doing something completely new, you just guess. Make the tasks small enough to handle and usually the good and the bad estimates add to a good average that works.

Frankly the estimate is one my least favourite parts of a project especially when the customer wants a delivery date and the total costs in advance.

Keep in mind that there’s a million different ways to “measure” software engineer productivity and estimates and it’s constantly evolving. To keep things simpler I’ll focus on modern SDLC (software development life cycle) processes that many teams use these days (You might hear the term agile a lot).

Generally speaking, larger problems get broken up into smaller ones. A lot of teams will try to break up the problem to a point where individual tasks ideally take only 1 or 2 days to complete. Obviously there will be larger ones but a good team will do everything they can to ensure that no particular task ever takes more than a week or two to complete. The smaller and more specific the tasks are, the more likely you will be able to accurately guess how much time it’ll take for someone to complete.

Most teams will try and collaboratively gauge the complexity of the task. They look for things like risks. For example, does the task need someone with specific experience or knowledge? Is it possible that another team can prevent you from completing the task? Is it something that your team has never done before? Does testing the task require a lot more coordination? If so, pad the initial estimate until your team feels relatively confident that they can do it in the estimated time. The team will try and build a consensus that everyone on the team would be comfortable with and discuss outliers from the team to understand everyone’s though process and concerns.

As for how the team decides on implementing a solution, most of the time you have an experienced engineer lead a design session with the other devs. In these sessions they plan out how they will solve a task or a set of tasks and makes sure everyone is comfortable and understands what it is they’re building. If the project is large enough you might have someone more senior (architects) working with multiple team leads to lay out the larger picture that they will then take back to their teams to figure out the details.

If you’re really interested in it you can read about modern SDLC processes, there are literal textbooks written about it. Modern processes fall under the “agile” philosophy but there are tons of ways of actually doing it. You can take a look at things like SCRUM and Kanban to get a feel of how some of these work from a project management standpoint. But from an engineer standpoint, nearly all modern agile processes follow the spirit of what I described, even if the specifics vary based on the team. I’m being vague here because good teams will be flexible and adapt the process to fit their situation, skills, and needs. Which means the specific process any given development team follows will be different from others, even if they’re in the same company or have the same leaders.

For more standard software jobs you can break down tasks, and estimate how long each one should take. This is usually pretty basic since a lot of the tasks you are going to be doing have likely been done before in some way, so we actually know how long they should take. Your performance would then be based on how slow/fast you were getting these done compared to what was estimated.

However for me, I work in experimental tech. We have no idea how long anything will take, or if it’s even possible which pretty much puts me at the mercy of my supervisor judging if I did a good job or not. A lot of times I’ll spend an entire day pouring through and researching documentation trying to see if what we’re attempting to do will even work only to commit a few lines of code at the end of the day and worry that it looks like I got nothing done.

Many good answers have been given… The question reminds me of a book called The Mythical Man-Month by an IBM software engineer from the old, old days. In it the author says that over a long history of projects he has found that adding developers to a project slows it down. So even then with terrible tools, they really couldn’t measure software work in any reliable way. And they had no idea how to make it more efficient.

In the modern era, Microsoft is famous for having hundreds of working groups colliding with each other and re-doing their code over and over. The results speak for themselves.

Modern programming processes follow an Agile methodology, where you have a backlog of work and then perform the highest priority work during a set period of time. As a result of this, there IS an objective measure of performance. The individuals that can take on tasks from the backlog and consistently deliver them with the highest level of quality (passes the testing and definition of done), over and over.

With this approach, the consistence is the measure. The ability to complete more tasks per sprint is a measure. Consistent grooming of the backlog and management of the tasks, while minimizing rework, is also a measure. If you have 5 people who do all of this, consistently, over time, you’re very fortunate and each of them is an equal contributor to the team and solution.

That being said, the next measure for software engineers then comes to experience with languages, middleware, toolsets, data storage options, cloud services and applications, and other elements. If I have the ability to write a web application, using cloud storage, in a secure fashion, in a week, to demonstrate functionality for a customer/leadership, then this is going to be recognized as having greater inherent value. As a result, growing your skillsets and the ability to incorporate/cover more niche products and elements of a broader solution, without needing ramp time, is going to make you more valuable out of the gate.

At my workplace, we aren’t rewarded for doing our day job. Being a good employee is expected of us to get a paycheck and average bonus. What gives us a greater bonus and recognition consists of three things: Developing yourself, Contributing to the Business, Contributing to others. This means that you do your day job, but you should be working on yourself, doing something that improves our business, and doing stuff that elevates our peers.

So those 5 engineers would be seen as good workers, but meeting expectations of their day job. The engineers that have a measurable contribution to their peers (presentation, training, mentoring), or to the business (template creation, reuse, engagement expansion), or themselves (learning new X or Y for future use), would then rise above the others who are just doing the day job.

In short, there are objective measurements under the right coding conditions. If you’re just a warm body and developing software in an unstructured manner, then yeah…it’ll be pretty subjective.

The preferred path is

1. Architecture: Determine business processes, functions, people and what they are trying to accomplish. “Domain” architects for things like I.T., Data, and Applications each plan out the big picture on how their domain will need to change to be able to support the vision.
2. Design: The groups begin to break up their respective change areas into logical blocks – these people support these things, this group owns this application, this team stores data. Those groups then look at the what the needed inputs to the larger architecture – oh, they need sales transaction data in order for their plan to work; well, we don’t actually collect all of our sales data the same way. Lets make a plan on what standardized sales data will look like
3. Development: You now analyze your respective systems to determine what changes are needed to create the standard data set. One change might be to update the UI on sales registers to all transmit sales data with a $000.00 format, as some only have $000.0 format. However, we know that rolling out a change to the registers at over 2000 locations will be a big change in the sense of regional testing, integration with the financial system, integration with the data repository, training, communications. Thats too much for one person.
4. Delegation: The UI change will be easy, as its invisible to users – the button now sends 2 decimals instead of 1. However, their SALES numbers will show up on their screens and in-store reports differently every day now. You need to verify every variation of every screen to make sure that the screen can fit the change and not cut out zeroes. Well, thats that just time consuming – we have 87 different regions and logging in, making the change, making sales, and verifying the UI and reports will take about 2 hours each – thats roughly 180 hours, though you’ll get faster. thats an entire month for just 1 person and so we probably want to cut that down to 2 weeks, so lets have 2 people doing that. Meanwhile you still have the financial system, the reporting server, all of the reports, etc to send the data to. We want to say “Well. some systems already send the .00 so it should be fine”. That is a big no-no, never assume technology will work. Someone may have defaulted the .0 senders to *automatically add the second 0.* That would OVERWRITE any value you sent at the second decimal. People do that crap all the time. There are 43 sales reports and we need to verify the reports. Those will take about an hour each. So lets assign that to someone. We now have 2 people working 2 weeks and one for 1 week.

As long as you understand what you are working towards, and all of the things that are in play, you can put metrics around the quantity, complexity, unknowns/risk, and time associated with each one. Those are called SWAGs – Scientific Wild Ass Guesses. Youre usually wrong, and there is always something you didn’t think of so all of you are overworked all the time and things are rarely on time or under budget.

[removed]

There is not a single accepted answer to this question. There are many competing methods, collectively called “software development methodologies”. Here’s a short list of popular software development methodologies:

1. Waterfall
2. Agile (there are derivatives like Scrum, Lean, and XP)
3. RAD (Rapid Application Development)
4. FDD (Feature Driven Development)

As an agile product pwner, I won’t answer your question but rsther ask you: why?
You can try to measure your software engineers but what’s the point. More likely then not if you think your SE are underperforming there is a problem, usually not laziness. It’s very likely that you don’t want to fire anyone (because that’s paperwork, painful and you’ll need to hire someone else which gives no guarantee to be a lot better) so your second best option is to understand what is wrong and then help your team skillup.
Happy devs who have time to learn eventually end up being good devs. Pressure is almost always useless

I manage software engineers. The truth is that there is no way to fairly judge engineers with “objective” data. Comparing individuals or stack ranking is a toxic fools errand.

There are far too many variables. For example how to you judge a generalist v.s. a specialist? There are 100 other dimensions and each one is fuzzy and requires bias checking at every stage.

Good companies try hard to do fair and wholistic evaluations to ensure pay equity but only in fairly wide bands.

It’s far better to support growth than get hung up on evaluations.

Very vaguely. Big problems are broken down into small pieces that one can hopefully get done in a few hours or days. But sometimes there is a big problem, and more time is needed.

In an Agile system coders are constantly estimating how long tasks will take and updating them with how long they actually took. From this you can make better estimates in theory. Managers can use those estimates to get a general idea of when bigger projects will be done.

Problem is, few people can plan ahead for the problem they did not think of. Development is one long string of problems.

It’s mostly based on our own estimates and how well we meet those estimates. But it’s also measured against our peers’ and leads’ estimates. If we consistently estimate higher than other coders (and we take longer than other coders) then we’re probably not being as productive. It’s tough though – one of the hardest things to learn as a software developer is how to give time estimates for something which you don’t know much about. Like – I’ll be asked to estimate how long it will take to add a feature to a program when it’s not a program I’ve ever worked with, it’s not a business process i’m familiar with, and it’s not a customer I’ve worked with before. This kind of thing, to some degree, happens frequently.

As a young coder you often start out making bad estimates, because you know what it’s like to do something from scratch so you use that as your basis. Maybe you need to add a field to a webpage – nothing complicated. So you say it’ll take half a day, figuring it should only take an hour so you’ve padded a full 3 extra hours in. But then you look and you find that the page is being automatically generated from some other code. And that code references a database view. And that view is a complex reference that filters out based on a myriad of factors. Plus the layout of the page is standardized using some templating, and changing the template to accommodate the new field will break other pages. And none of this is documented, in fact it took you two hours just to figure out what tables in what databases were involved, and whether or not you had the permissions. You’re also unsure which environment is right, the code points to (local) and the dev environment points to dev, but the test environment seems to point to both dev *and* test in different connections and the layout is entirely different. The code that’s in production doesn’t seem to match anything in the test environment or the dev environment. Now you’re 3 hours in and you haven’t even started adding the field.

Then you end up over-correcting. Suddenly your boss is worried why you think every minor change is going to take at least a week. You keep talking about “unknown unknowns” as though they’re the boogeyman. You question whether time is real.

Eventually you end up where I am now (I’m about 12-15 years in, depending on the measure) where you have a good sense of what the factors are that will go into the estimate – you can give me that same “add a field” scenario and I can tell you it’ll take me somewhere between 2 hours and 2 days, depending on what the code is like and whether there are unknown problems that need to be fixed first. Particularly cases where (and it happens more than it should) the code that’s deployed on production doesn’t match anything we have in source control and we’re going to need to reverse engineer it with a decompiler.

So as I get more experienced, my estimates get more realistic in both directions and they also are able to come with contingencies – I can tell you not only how long it’ll take but I can tell you why it could take longer, and when I’ll know how long it will take.

This is an old school answer – I retired before so many of the methodologies mentioned here were in place. On large projects I worked on, people were assigned larger pieces and asked to estimate what it would take to do the work ( as opposed to current methods where small pieces are assigned and a team does a collaborative effort). Some people’s estimates were just awful, either because they wanted to look good at the start, wanted to fit in to what they thought others expected, or were just bad programmers. Some were spot on, within days on a year long project. Eventually, assuming a stable team, it became pretty clear how to weigh estimates. This was all product development, not time sensitive customer or contract work. [Warning: bitching ahead] Eventually, the whole industry changed, and specifically my company’s management changed and schedules became more important than quality. At that point, lots of those scheduling methodologies like “Agile” came in and just made the job harder. The bean counters, collaboration weenies, and micromanagers won and quality suffered. Glad I’m out.

Largely engineers are measured on results, few people care if we are lazy or working 80 hours, results are that count.

Either by asking you questions about coding that are hard to answer without several years experience or through a dumb test:

The industry picked a few small, easily memorised tasks that have nothing to do with the job you have to do and ask you to do one of them at an interview. It’s like doing 2 questions from a high school geography exam before being accepted as the civil engineer in charge of a huge bridge. Sure geography is slightly related, and if you studied it 8 years ago then the question will probably still be in your head, but it is a laughable proxy for the skills you will actually need.
Tasks include: implementation of an algorithm for shortest path, sorting a list of numbers, solving a logic puzzle or manipulating a data structure in a pointless way.

All of these are about high school level maths difficulty, have been done before and can easily be found online, but are impossible to complete in 8 minutes while the interviewer looks over your shoulder at every line you write unless you already know how. They will gladly fail you if you don’t finish the task even if you have a credible CV that marks you as a senior developer with a ton of experience building much more complicated systems.

In summary, they don’t, but if they do, they do it badly.

Generally the person assigning work is a senior developer of some description – that’s not necessarily their job title, but they’ll be an experienced developer themselves

Since they’re relatively experienced they have a rough idea of how long things take. They’ll also have a reasonably good idea which tasks are most at risk of delays and will endeavor to spread them around the team so that one person doesn’t get hit with them all and end up way behind schedule.

A good project manager will then add some slack to this timeline, and then the developers start the work

Then if it takes longer, the developer informs the project manager as soon it becomes likely and they adjust timescales, manage stakeholder expectations etc. if it’s faster, the developer just starts the next task early.

With experienced team members doing the assignments, managing expectations etc, it’s usually pretty smooth unless unrealistic expectations are being dictated from further up the tree

they aren’t, especially if you aren’t a man or white. bias in software engineering is a well-known issue, and many women end up doing a lot of the “glue work” to keep the team running (see https://noidea.dog/glue), get penalized for not being “technical enough”, and get undervalued as a result. many times people who are underrepresented in tech get worse/less visible responsibilities and then penalized for not doing challenging enough work. technical interviewing is also subject to bias; many interviewing techniques assume tons of free time (not good if you have a family or chronic illness) and a traditional college CS education. they’re also shown to induce stereotype threat in underrepresented groups and misjudge their performance anyway. of course, since people in this field pride themselves on being objective, they insist that they are being rational and selecting the best talent in ways that couldn’t possibly be biased. this is a lie.

tl;dr they aren’t unless you’re a white man lol