> When was the last time you had to debug an ancient codebase without documentation or help from a team?
All the time. 300-400k SLOC in C++. Legacy in the sense that there were no tests of any kind. Little-to-no documentation. Solo developer at the tiny company. Fix bugs and add features while keeping the system available to the tens of thousands of users.
A more recent example: here’s a patch for a critical feature we need. It was written over a year ago. The original author isn’t available anymore. You can write the code from scratch or try to resurrect the patch against master.
Being able to jump into a project and lead people towards some goal is definitely a skill for senior developer positions. Yes, you generally have a team you can lean on and have the ability to do research and all that. But how do you show that you can do all that in an interview?
Agree with the conclusion that a good thing to test for is for problem-solving.
The tech side depends a lot on what you’re doing. Although it gets ridiculous and organizations get lazy with this part. You don’t need to be white boarding graph algorithms for a junior web developer role. If your application is a social networking role and you’re interviewing a senior developer or architect? Definitely. They’re going to be teaching this stuff and need to understand it at a deep level.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Heck, I have to debug the stuff that some idiot (me) wrote six months ago and it might as well have been someone else who wrote it for all I remember about how to debug it
So many times I'm reviewing code and really wish unpleasant words with the developer that created this horrible code. Problem is, people think I'm weird when I talk to myself like that.
Edit: apparently this wasn't written clearly enough that I was the origin dev that I'm having words with???
When is the last time you’ve been trying to get help from another team, only to realize that the engineer you are talking to can’t write code, even with the AI help?
+1 for "all the time". Today I have been debugging a critical piece of the system which is written in Python (none of the rest of the system is) and largely hasn't been updated since 2020 and, you'll not be surprised, has no comments, no documentation, and a fucked up deployment system which makes me cry every time I have to think about it.
Last week I was debugging some similarly uncommented, undocumented, Go code from 2020 written by lunatics that is also a critical piece of the system.
Yeah, I feel like jobs that don't require you to reverse engineer a bunch of stuff are the exception, not the inverse.
Hall, I do greenfield embedded programming. Most of the code I touch is completely new; the entire codebase has been replaced and rewritten by me over the last three years. Not one other developer has contributed any significant amount of code. Even then, in this scenario, I'm still reverse engineering some arcane undocumented code at least once a week. Nobody ever documents their libraries properly so I have to rip everything apart to see how it works. Or I need to adapt some vendor example code or a GitHub repo into a slightly different form. Even just existing in the ESP-IDF environment I have to dig into the core code to figure out what the hell it's doing and where these error messages come from.
If you can't read someone else's Gordian knot of undocumented code, I'd argue you are probably not very good at writing code. Reading bad code is a vital part of learning to write good code, and is generally just a fundamental core skill that every programmer needs to have.
+1 here too. The situation always comes up after some kind of fucked up acquisition. Nothing like inheriting an entire code base built on someone else's stack when all the original developers decided to quit or were laid off. Ancient versions of Python and Java, deployment done by pull and pray. Multiple versions of the same service in different places because they were half way through some modernization when it all came to an end. Fun stuff. Getting your bearings fast is a real skill.
People who don't have this problem, are people who treat the their dependencies and the lower layers of their stack as ossified black boxes, rather than something they can reach into and fix.
IMHO the willingness to "dig your way down" to solve a problem in the correct layer (rather than working around it by adding post-hoc ameliorations in "your" code) is one of the major differentiators between junior and senior engineers.
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
> Fix bugs and add features while keeping the system available to the tens of thousands of users.
Don't tens of thousands of users warrant more developers? Or having enough of a budget to work on tests, or other things to improve the developer experience and be able to work without lots of stress? That's unfortunate.
Don't understand one thing: I can read any code of any legacy code base, sure. What I cannot do is to asses if the code is supposed to do its job (e.g., the business logic behind it). Is the code I'm reading supposed to calculate the pro-rated salary according to the "law"? Just by reading the code, you cannot know that. You need help, either from other developers who perhaps know the codebase of from product experts that know the business logic. Or documentation, sure, but this is usually a luxury one doesn't have.
So, definitely, I always need help debugging any kind of code (unless it's rather code that doesn't deal with product features).
> > When was the last time you had to debug an ancient codebase without documentation or help from a team?
> All the time.
So you were forced to do it in a silo? No access to any type of documentation, whatsoever? Not able to refer to any C++ manuals or any other type of documentation? Not permitted to access the internet to look things up? Barred from using online forums to seek help?
Yeah I thought that was a weird thing to highlight. The "debug this!" kind of interview is pretty much my favorite, because it's by far the closest to what I do for like 90% of my time at work (when I'm not doing communication tasks...).
A partner of a friend quit their job earlier this year. They then took 4-6 weeks to prepare for each interview with Big Tech companies (4-6 weeks for Meta, 4-6 weeks for Stripe, etc.). Along the way, they also took random interviews just to practice and build muscle memory. They would grind leetcode several hours a day after researching which questions were likely to be encountered at each Big Tech.
This paid off and they accepted an offer for L6/staff at a MAANG.
Talked to them this week (haven't even started the new role) and they've already forgotten the details of most of what was practiced. They said that the hardest part was studying for the system design portion because they did not have experience with system design...but now made staff eng. at a MAANG. IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Wild; absolutely wild and I feel like explains a lot of the boom and bust hiring cycles. When I watch some of the system design interview prep videos, it's just a script. You'll go into the call and all you need to do is largely follow the script. It doesn't matter if you've actually designed similar or more complex systems; the point of the system design interview is apparently "do you know the script"?
Watch these two back to back at 2x speed and marvel at how much of this is executed like a script:
Back when I had a job hiring people we created problems we could walk people through and see what they figured out on the fly/what they knew but didn't know they knew. That was what I was taught whiteboard problems were, not this lame leet code. But I grew up with both parents in 1980s/90s Santa Cruz tech. The current scene adopted the practice but made it exclusive when it was intended to be inclusive (because there wasn't a huge talent pool back then so they were looking for people they could grow into developers).
One of my hires was a physicist who didn't know any of the jargon and had a look of terror when she first started on our whiteboard problems. Once we led her to a comfortable spot and she got into it she started talking about tools she made to help with her physics work with zero dev knowledge (she was shocked when we called her tools software, she was like, but it wasn't REAL software, we were like hate to break it to you, but that was not only real software you wrote, but crazy impressive). The white board problems were a tool to highlight potential.
I get that these big companies just want drop in widgets not people and that is what they are searching for, I just think it's funny they are using something that was created for the exact opposite purpose.
Part of me is kind of glad I'm too old to be one of the cool kids and have no hope to land a job that does these sorts of code challenges.
Well it shows that you are ready to invest a lot of your own free time into being recruited there - i.e. you'll be a loyal and hard worker. And that you're smart enough to learn it all. So I would say it has a character + IQ component to it.
Of course 95%+ of it will usually be useless in your real-life work, to solve most of those problems in the time given you need to know the problem and the optimal solution to it so basically memorization with little thinking.
If you talk to hiring at some of these companies, it is intentionally designed to be this way so that it is fairly meritocratic.
In other words, anybody, regardless of what university they went to or what courses they took out what advantages or disadvantages they had, can learn this stuff in a couple of months if they have what it takes for the role. And because the skills are so standardized, the process is pretty differently objective.
It's not expected that they'll actually use the specific skills in the job. But it shows they can learn skills of that type and then perform them at a high level in a stressful interview situation. Which is a great signal for whether they can learn the skills needed for the specific project they wind up on and perform in a high stakes deadline scenario.
I'm not defending the system, but I am saying there's a clear logic behind it.
The thing I think is funny in all this is that hiring a new manager is fraught with a high amount of risk (more so than an engineer), but they don't have nearly the level of hurdles to get over. Does the company interview past employees of the manager? Did the guy have alcohol, drug issues, or weird sexual things he did to his direct reports or others?
> It doesn't matter if you've actually designed similar or more complex systems
You know you've designed more complex systems. The interview has no way of knowing if that's true, or you're an actor following the "I've designed complex systems" script.
And acting talent is arguably more common among the population than programming talent.
> IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Also they say they've forgotten. But if they were able to get that position, they probably could do medium level leetcode problems. So, I'd doubt they've forgotten all of it. I'm pretty sure they'd still be able to solve easy level problems, which most people can't solve off the cuff. They also probably still know the complexity of a bunch of essential backend operations (search, sort, array and hash lookups, tree & graph traversals, etc).
No home study grind on puzzle problems is a substitute for years of practical experience, which is what most teams actually hope for in their higher-level engineers.
The point is not that the friend didn't pick up some implicit knowledge or become a sharper engineer than they were before grinding, it's that by exploiting the screening strategy, they got placed into a job they're not truly qualified for.
Are they bright enough to fake it until they make it? Maybe, but that's not going to be the case for many of the countless placements that were made like this, and hints at why both product and software quality is in bad shape these days.
Yeah, in some ways I like "system design" type questions, because at least it's pretty creative and more like what I do at work than writing some little algorithm with a custom data structure.
But on the other hand, it's also not a very strong signal for my years of experience working on and designing portions of systems. I've never actually done "design a system from scratch", because every system that is big and complex evolved from some smaller and less complex system.
So the kinds of questions that come up in these system design interviews are best answered in real life with "I dunno, I'd pick something simple and refactor it later if it becomes a bottleneck".
"What kind of database would you choose for this?" "Whatever we're already using in production for other things." "How would you model this data?" "As simply as possible, so that we can get it out fast and start learning things."
But these are not interesting answers in these interviews. I have usually found myself saying "well, in real life, I would advocate for simplicity and iteration, but for the purposes of this interview, I'm happy to discuss potential trade-offs in technology selection, architecture, and data modeling".
But I think it's easier to study up on the "right" answers to the architecture and modeling questions, without spending a lot of time learning how to start simple and when to evolve toward complexity, etc.
I think the context is important in the question. I’d argue that a system design question is optimally answered differently depending on the size of the company and the scope of the internal tech stack.
Doing more of the design up front is important in a big tech environment almost all the time because:
a) many different possible infrastructure (eg. databases) are already at your fingertips, so the operational cost to using a new one is a lot lower. Also, cross-organisation standardisation is usually more important than this marginal extra local operational cost.
b) scale makes a big difference. Usually good system design questions are explicit about the scale. It is a bit different to systems where the growth might be more organic.
c) iteration is probably way more expensive for production facing systems than in a smaller company, because everything is behind gradual rollouts (both manual and automated) and migration can be pretty involved. E.g. changing anything at all will take 1 week minimum and usually more. That’s not to say there is no iteration, but it is usually a result of an external stimulus (scale change, new feature, unmodelled dependency etc), rather than iteration towards an MVP.
Now, a lot of this is pretty implicit and it is hard to understand the environment unless you’ve worked in a different company of the same scale. But just wanted to point out that there is a reason that it is the way it is.
> No home study grind on puzzle problems is a substitute for years of practical experience, which is what most teams actually hope for in their higher-level engineers.
Perhaps.
I had 9 years of practical experience myself before going through that grind (leetcode + system design) some years ago. It is no substitute, but you don't come out of it the same engineer. You read, write and understand code differently.
Also, I have to admit that with the new perspective, there's a reason why I'd probably hire the newbie who's done this over the experienced engineer who hasn't (and maybe continues to dismiss it). The two gaps are not equivalent. In my experience the newbie will probably cover many of the necessary bases on the job fairly quickly with mentorship. I can also appreciate that 9 years of practical experience didn't help me cover DS&A or system design. I doubt that any kind of mentorship would've helped me either. Learning this stuff is a grind, plain and simple.
A newbie will also make mistakes that I can understand. I can see them pick a heap, or a linked list because it's the algorithmic sound choice for the problem. Then I can point out that for the specific use case, an array might not be as efficient, but it's a good trade-off to simplify the code. They will understand this. But an engineer that indiscriminately reaches for arrays and hash for any situation is a different story. Exponential inner loops for big searches, huge recursive calls. Those things show up in code written by experienced coders IRL. You click on a button and wait 5 seconds for the response.
I'm not sure who's faking it until they make it, but based on my own experience, I'd say that perhaps I successfully faked it for 9 years.
In this example, we're talking about an L6/staff role.
If the role is correctly staffed, nobody should have to be looking over that person's shoulder questioning data structure choices and algorithm efficiency regardless of what the person's background is.
It's absolutely the case that there are people with 9 years of experience who are not prepared to meet the needs of that role. There are people with 19 and 29 and 49 years of experience that fit into that bucket. Practice alone doesn't get you there.
But as you note, as a rule, you expect your bright, leetcode-grinding "newbie" to need mentoring and oversight to compensate for their propensity to rely on shallow theory where they don't know better from experience. That's not someone a team wants in a L6/staff role either!
For high level roles, you need bright, curious people who are experienced but not rigid. Admittedly, there's only so many of them, but when you start having to fill those roles with people that don't meet each of those criteria, you can't be surprised when the team's morale and output suffer. Which is exactly what we see as an outcome of the recent boom (and each prior one), because you end up having the blind lead the blind through the minefield of engineering.. and that's going to be messy no matter how clever or book-studied those folk happen to be.
That’s a false dichotomy. I’ve never felt the need to do the DS&A grind for one of these roles and I would never dream of writing the ungodly inefficient code you just imputed to my peer group.
> Exponential inner loops for big searches, huge recursive calls. Those things show up in code written by experienced coders IRL.
The thing is, those things are easily fixable after the fact. A bad architecture is not, and without practical experience designing systems in the real world, you don't have those instincts to guide you towards the correct choices because you haven't made any mistakes to learn from.
You may remember these details from the GP comment and my own:
> L6/staff
> higher-level engineers
Traditionally, one would work up to senior roles over the course of one's career, often by pursuing internal opportunities to learn and exercise new skills within one's current role before applying for new roles that rely on those now-practiced skills.
Placing into senior roles because you did well at Stanford, spent a couple years writing CRUD apps, and then grinded puzzles for a couple months is inevitably something that happens when the industry needs to fill more senior roles than there are engineers with suitable experience, but it's not a healthy phenomenon and definitely shouldn't be treated as the norm.
> Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Having done this prep, I can't tell if this comment is sarcasm. Building real systems makes you a good engineer. Maintaining systems over a long period of time makes you a good engineer. Working with other experienced engineers makes you a good engineer.
Doing this prep you do learn a few things along the way, but it's contrived. You already know what you need to learn, which is often the hard part on the job. It's works as a filter since the ability to learn concepts is important, but it's usefulness continues to trend downwards as it becomes more standardized.
A moderately smart person was selected for a good job perhaps over many many better possible hires simply because that person had the leisure to learn the game. Inefficient. But nice for that individual, naturally.
But is there a good way to find the "better possible hires" which doesn't have other significant disadvantages? If you have a convincing method of doing that, many companies would be interested in your ideas.
In the US this doesn't work well outside of college internships. Most tech workers don't want to shift to a new employer with a probation period. We already live under an "at will" employment relationship so employers can let you go anytime and workers can leave anytime. To have real value to a probation period for workers, we have to guarantee its harder to fire you past the probation period.
This will strongly favor people who believe that that they cannot get a good job, because if a candidate has multiple offers, why would they choose probation vs regular job?
(Note I am saying people who "believe" they cannot get a good job. This would be people who worry a lot, people with unusual experiences that other companies avoid, and under-performers who got fired. I am sure there will be some great hires in those groups, but likely less than during regular hiring)
Are you suggesting that any coding interviews and challenges are simply removed from the existing processes? That just means you end up with more candidates to choose from, which doesn't sound helpful at all if your goal is to end up with better hires as the comment above was suggesting.
Yes, ask give interview tasks which are realistic depictions of the actual job tasks.
No, the hiring managers that are into cargoogle culting are not actually that interested in how to do interviewing properly. Not unless, say, google does it and they can copy it.
For them the important thing is that leetcode is a safe, defensible choice because "everyone else does it that way".
One of the main questions I ask in interviews is basically "we have a data pipeline with goal X and constraints A, B, and C. How would you design it?" Depending on how they do, we'll discuss various tradeoffs, other possible goals/constraints, and so on.
This is based on a real system I designed and have been maintaining for ~5 years, and is also very similar to other systems I've run at previous jobs.
About half the candidates complain that it's not a realistic question.
> Yes, ask give interview tasks which are realistic depictions of the actual job tasks.
I think this is exactly what a lot of companies try to do when interviewing. Depending on how much time they want candidates and interviewers to spend on the task, this ranges from leetcode-style problems to bigger coding challenges (possibly with some debugging or collaboration involved).
I don't like the "state of the art" of tech hiring but I think this thing you say can be said to any of us in our jobs and that's not fair.
We all part from the same place (understand me: from the standpoint of the hirer, I know we have different vital stories). They open the position to all of us and want to be impressed. The one that makes it, gets the job.
It is the best way? Absolutely not. Is it unfair? No.
It’s like the bar exam for lawyers. It bears absolutely no relationship whatsoever to the actual job and the work you will be doing. It’s a pointless ritual. And the point of the above story is that the person performed the ritual and then promptly forgot all the words once they succeeded. It illustrates the pointlessness.
I was a lawyer. I passed the bar exam (first try, no less). I practiced. The bar exam does none of what you claimed for it. Now, I offered it here as an analogy, but I am not a software developer. It just sounded akin to what I do know from my own experience and from talking to literally hundreds of other lawyers, law school professors, judges and their clerks. So maybe it’s not a good analogy to software development. I don’t know.
> It's not a pointless ritual. It tests for determination, grit, and willingness to grind on difficult, frustrating and ultimately value-free tasks.
This is what people say whenever one criticizes the methodology of any test. They say that the real test was actually the friends we made along the way. The real test was the degree to which you are willing to humiliate yourself, or to accomplish things you don't understand the purpose of, or to accomplish things that you do understand the purpose of but disagree with logically or morally.
We filter for the worst, most damaged, and most desperate people.
> We filter for the worst, most damaged, and most desperate people.
These are probably the best candidates, from the perspective of a manager who is concerned with stability and not rocking the boat, at the cost of results and workplace quality.
"Seemingly smart person getting a good job" is a criteria for filling headcount during a boom, but regresses the industry's maturity by placing inexperienced people into critical roles.
The problem with this story is that people need to work with and rely on this person for responsibilities they're not yet qualified to meet. In some cases, a "smart person" will be able to grow into the role, but the road there is long and messy, which becomes a frustration for colleagues, supervisors, clients, users etc.
Because of the prolonged boom we just went through, the industry -- especially at FAANG's -- is now saturated with smart, naive people trying to fake it until they make it, leading to a gross decline in quality and consistency compared to where we have been and might otherwise be.
It took forever. Literally tens of thousands of dollars of opportunity cost, with a lot of risk of having nothing to show for it.
The bummer is that you're right, it actually is worth even this much investment, because these companies do pay extremely well. But it's still horrendously inefficient, because the companies are getting a very small improvement to their signal to noise ratio, at this great cost (which, notably, they don't bear).
Yes, and they should also be able to understand when the system that creates those smart but speculative bets is creating deadweight loss, and identify that as a problem.
BTW, I have personally made this bet in the past (though I didn't put 3 to 4 months into it...) and it was indeed an excellent bet to make. But benefitting from a system does not preclude identifying problems with it.
although its a different paradigm that I’m used to, I agree with this general concept
if you provide utility to the market, you shouldn’t need credentials or a corporate ladder to climb to prove it, it should be immediate compensation no matter how disparate
so despite how coveted tech compensation packages are at this tier of company, a seemingly smart person getting a good job after doing a contrived aptitude test does meet that criteria. other people outside the field (and within) have difficulty passing it and don't have the cognitive ability to study for it, or the financial stability to prioritize studying for it
I also agree that a less contrived aptitude test would be better
Apparently they expect people to work for free for more than a month to learn material they then won't use.
In my mind that's a rather nasty practice.
Not that I'm complaining. I'm happy to pick up people that are good at computers but wouldn't be able to pass that hurdle, and probably wouldn't hire anyone that has.
If you're getting a PhD solely for a job in marginally related part of industry, that's on you. The experience is useful if you're doing it for right reasons, instead of as a proxy ritual to boost your CV. There are more efficient ways to do the latter.
Is "cramming" the same as "learning"? Is knowing the chemical reaction between an acid and baking soda the same as being a world class patissier? Many of these exercises are like "demonstrate that you can make this dough rise" but not actually considering "can you make an artful and delicious pastry"; the two are totally different.
The most surprising thing when asked about their experience was that most of the details have already been lost before even starting the role.
You can only get so far as a pastry chef knowing the recipes. Understanding the chemistry does take your craft to a higher level.
I’m never really surprised that there are so many security vulnerabilities out there. Tons of software is completely half-assed: slow, error-prone and inefficient. The market supports it because VCs keep pumping out companies that throw spaghetti at the wall until something sticks. And it has had an influence on the kinds of skills we optimize for at the hiring level.
The problem I see isn’t that we’re trying to hire better developers. It’s that the tests we use are misaligned. Getting into a MAANG with all these hazing rituals and then you get stuck resizing the corners on buttons or editing XML configuration for some ancient system is a waste of time.
But so is hiring programmers that only understand frameworks.
At some inflection point, though, cramming becomes learning. This is in a different place for each individual, but it's true regardless.
Nobody hires anybody other than artists based on whether they can "create an artful and _____ ____". People are hired based on either what the have done, can claim to be able to do, or whom they know.
Depends on the situation and job of course, but cramming isn't necessarily always a negative thing. Sometimes you may only need to learn a technology/skill/etc. to accomplish something quickly, but then not need it again.
But yeah in general I agree with you – I just don't think it's necessarily always so clearly a waste of time.
I took a similar approach -- took 6 weeks off doing interview prep, then concurrently applied to three FAANG employers. Got offers from all three, the worst of which was a 36% raise over my former non-FAANG job.
The problem is not everyone has the chance to work on super-scale systems. So what are you going to do? Weed out anyone with the potential to become great at that just because they've never had the chance to try?
Demonstrating the ability to learn, even if it's just scripted, should be good enough. Granted, they might get passed over by someone with actual experience, but the lack of real experience shouldn't be a deal breaker. It's fine as long as the employer has realistic expectations of the employee.
Companies aren't incentivized to eliminate false negatives in hiring (talented people getting overlooked), they just need to make sure there are no false positives (bad hires). I'm guessing it's probably rare for companies to think about the bigger picture, like the long term health of the economy and nation, where it's actually really bad if there are a bunch of really talented people that are being underutilized.
If I were interviewer, I'd probably do a coding assignment that builds on an already existing mini codebase, quarter or half day assignment (2-4 hours), with a strict time limit to turn it in by the end of the allotted time. It would test for someone's ability to read unfamiliar codebases, with the codebase being reasonably small for the little time that's available, and the ability to build features on top of it, to write clean readable code compatible with the codebase, etc. And an a 30-60 minute more open-ended informal knowledge interview to gauge how knowledgeable the candidate is and where their strengths and weaknesses are. And previous work would probably be a good indicator, like open source work and/or previous job experience.
I recently ran an interview process for a relatively senior eng role at a tiny startup. Because I believe different interview methods work better for different people, I offered everyone a choice:
1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.
2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."
3. Submit a code sample of pre-existing work.
Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.
I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!
Interesting. I think I slightly prefer take homes, mostly because they're not as hectic. The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. At least with an in person, I know that the company had to incur a significant cost so I know my time is less likely to be wasted. There's already a growing asymmetry with job searches, and this is one more trend that accelerates that. I still have a non technical friend who's annoyed with me for not completing a take home years ago, because, while the main ask was obviously trivial, there was enough slop in the directions that I could kill hours trying to decide if something was good enough to submit. On top of that it was expecting some fairly narrow set of techs, that, while not difficult to pick up, were not something I was interested in trying to learn enough about for a chance to maybe help some company that "couldn't find any technical talent".
I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
The code I've written in personal projects have been pretty messy -- whereas code I've written professionally tend to be better (not perfect), but of course I can't share that code.
On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
Interesting! I like the idea of choice, but as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? I find trying to compare candidates fairly to be quite difficult, even when they have the exact same interview.
Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
> as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably?
That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring client, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.
Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
Highly agree with all of that, perhaps save the conclusion. I still try to standardize the interview process, but we have enough different kinds of interview phases to capture different strengths and weaknesses of candidates. I still want the interview to be fair even when people respond very differently. You’re right that it doesn’t often come anywhere close to a tie. But sometimes candidates aren’t vocal and don’t vouch for themselves strongly but they are great coders, and sometimes people are talkative and sell themselves very well but when it comes down to technical ability they aren’t amazing. The choice of coding interview could obscure some of that, so mainly I include other parts to the interview, though I’m still interested in how to compare someone who does take home coding to someone who does the live pair programming, for example. I kinda want to see how candidates handle both. ;)
in one interview for a python job i was allowed to submit solutions to tests in pike which i had more experience with, but none of the devs at the company had ever seen. for python they had an automated testsuite but i remember my interviewer telling me that the devs were pouring over my test results to verify them manually while we continued other parts of the interview. i got the job.
Why would you even do any of that for a senior role? I wouldn't waste my time with it, and it shows you don't know how to interview/evaluate for a senior position.
Why wouldn’t you? How should one evaluate a senior position, in your opinion? Are you suggesting a senior dev shouldn’t need to demonstrate any coding skill? (If not, why not?) Or are you suggesting that there are other faster ways to show coding skills?
I responded to a different comment but it's also relevant here.
You're hiring for a senior role, not a junior or mid.
Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.
It's like asking a doctor to go over basic human anatomy.
You’re underestimating the variance in senior devs, and you’re failing to look at this from the interviewer’s perspective. Sometimes doctors should be quizzed on anatomy, btw.
They need to be able to compare candidates, and senior devs will be expected to code. Their goal is to pick the best candidate. Asking them to demonstrate their coding skill isn’t just fair game, it’s what half the interviewees actually want. A lot of people prefer being evaluated on code than on soft skills (just read the threads here for tons of evidence.)
It’s useful to know whether a candidate is going to be on the principal engineer path or management path. It’s useful to know whether they are actually good at coding, and how good exactly. And it’s also useful to know if they are willing to do what needs to be done once hired. Someone interviewing for a senior coding position who refuses to code during the interview is about as big of a red flag as you can have, and I’ve been part of hiring teams that will politely excuse someone for that, and I agree with the reasoning.
This might not make sense until you’re on the hiring side of things, but having a resume does not entitle one to being hired for a higher title and more money.
I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
I'm a senior developer. Of course I know how to code. Do you know how many code challenges I've done over the course of my career? 99% of them where terrible at evaluating my ability to code. They are a waste of time at the senior level. You are much better off with a conversation, looking at past work and experience and references than you are doing code challenges.
After interviewing ~50 people using this technique - I'm sorry but there's no "of course". I cannot tell by looking at your past work experience or talking to you whether you know how to code.
I know this, because I used to look at resumes and have long conversations before I ran my pairing interview. And I found the results almost uncorrelated. Now I save the fun conversations for people who make it through the screening. But by then I already know if I want to hire them.
My pairing interview (it's a pairing session, not a code challenge) covers aspects of engineering that I specifically care about. It's not terribly hard. But at the end I'll have a pretty good idea if I want to turn this person loose in my codebase.
I've come around to the idea that all people who will be expected to write code as part of their job need to be evaluated for their ability to write code as part of the interview process. I've seen too many people write convincing and impressive resumes without a single ounce of technical ability to their name.
You're hiring for a senior role, not a junior or mid.
Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.
It's like asking a doctor to go over basic human anatomy.
>They already know how to code and have been doing it for years.
I'm telling you from my own experience that this is observably not true. Twice now I've been burned by internal hires who objectively had years of experience (easy to verify when they have been working within my division) and yet couldn't code their way out of a paper bag.
And tangentially, with some of the health care experiences I've had I very well might try to administer a basic anatomy quiz to potential doctors if I thought I could get away with it.
Because for a senior role you need to hire a person with senior role skills, I suppose. Just N years of experience at X company doesn't provide that. I'd rather look at someone's github repo than talk to their reference tbh.
Neither did I. But I'd rather just "smuggle" out some past work and "rewrite it" to make it inconspicuous, if I didn't have my own personal work to show. The company I work for isn't well known so saying "I worked there in a senior role for 10 years" isn't going to get me the next job. If I didn't have anything to show for that I either made for fun or could display from a past job, I'd not apply for another until I had tbh.
People say "No I don't have any past hobby work, never contributed to OSS, and all my professional work is verboten to show" that's not unusual, but it's also not an encouraging sign to me when interviewing.
The elitism is rampant. As an embedded sector guy everyone thinks every other RTOS isn't a real RTOS and the knowledge and wisdom don't translate.
Every time I find some open source I would contribute to, what I wanted to implement is already there or is already in the works. I'm not going to change shit just to change it. I'll buy them a coffee tip and leave.
Wouldn't that be a red flag if they don't have any public code at all as a senior developer? They aren't fresh out of school or just starting their careers. They should have something.
I'm a senior developer and my github is half guitar tabs. I'm not interested in peacocking. Maybe it's because Hackaday refused to put my name on the article with my senior design project years ago and I just don't want to play the game.
> These high-stress, solo coding assignments don’t reflect the actual job. Instead, they put developers in situations they’d never face in the workplace, where collaboration and support are standard. When was the last time you had to debug an ancient codebase without documentation or help from a team?
I have had to do this, on numerous occasions. Sometimes there is a system that no one who built it is left, the documentation doesn't exist or can't be found, and that's it. This is not a great example. Being able to debug a program, follow requests around and see what's happening is a good skill.
What this example is really bad at though is that you pigeonhole yourself into developers. If you have a legacy php application, you are weeding out the java, python, ruby, go etc developers that do have those skills but don't know php and will be slower at it. Despite them possibly being stronger developers given a couple months of hands on php.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
This I agree with. Suggested time can easily be gamed, so someone who has more time can look like a better candidate.
When I interviewed at Stripe, they had a "debug this!" question ready to go in a number of languages. (I think the list was something like: ruby, java, python, javascript, go, maybe one or two more.) I thought this was pretty great, but also seemed pretty labor intensive.
That is the way to do it, good on Stripe. Its definitely labor intensive, but I think that you either need to outsource it or be big enough that you can source the talent to build a half-dozen plus idiomatic and kind of big applications to make it a good test.
Yeah. In general I was impressed with Stripe's interview process. It was still the usual miserable day-long gauntlet, but with some thoughtful decisions mixed in, IMO. (This was 2022, FWIW.)
I'm pretty sure the thing they had me debug was a real bug they had hit in an open source dependency. It seemed like they had hit the bug, taken a snapshot of the code at that version, and written a test to exercise it. (And presumably then they fixed it, separately.) So it felt a lot like bugs I run into in the wild, but with some of the hard debugging work out of the way (getting it to the point of an easy repro in a unit test) in order to time-box it. I really thrived in this interview. It may be the only interview I've ever done where I felt like I was just using my actual skills like I would during the work day.
> This is like asking a Ruby developer to debug PHP as a test of flexibility.
Sounds like an OK test to me. Great (senior) developers should be able to do that kind of thing. Categorizing yourself exclusively as "a Ruby developer" is a career trap.
In the short term. And then the entire industry experiences some kind of a technological shift, as it will for many more cycles, and you're jobless as early as the first wave.
It's ridiculous how developers mindlessly accept that you should constantly be learning to keep yourself relevant, but keep it shallow by just jumping from one tool to another, instead of encouraging deeper knowledge of generalizable patterns that stay relevant across waves of technological disruption.
Exactly. I managed to ride a nice maybe 8 year wave as an "OpenGL expert" but the industry moved on and I didn't go more general with 3D graphics to climb out of my niche. I haven't been in that industry since the OpenGL waterline crested and went back out to sea. Learned that lesson.
Be a generalist. Yes, deep specialists exist and yes, some of them have successful careers based on their deep specialty, but betting on specialization is like a high school kid planning to be a MLB baseball player.
that's what i am trying to explain in every job application. but almost every job description expects multiple years of experience with a very specific tech stack. so far being a generalist with senior level experience did not yield a single positive response, let alone an offer.
Ruby is so god awful yet so many successful companies use it as the bedrock language there's always going to be jobs maintaining the piles of crap it leads to.
Bro this comment is so out of touch it's ridiculous. A 6 figure salary is lucrative. 7 figures is a crazy pipe dream that basically nobody will ever experience.
Isn't the average for software developers in the US ~100k(before taxes)?
Assuming that high earners are offsetting that to the higher end, most people aren't making 6 figures, and the bar isn't which language they're programming in.
One of the few areas of reliable statistics about US software developer pay come from the US Government Bureau of Labor Statistics. The median wage as of May 2023:
$132,270
This means half of all full time employed devs are higher, and half are lower. The mean is more skewed by higher earners but is similar:
$138,110
It also varies quite widely by geographic location, from a mean high of $173,780 in California to only $125,890 in Texas, from $199,800 in San Jose to $132,500 in Austin to $98,960 in rural Kansas (where I have actually developed software before!)
The short of it is, the vast majority of software developers do not make the top salaries. Even L6 is rare within the top tier of tech. There is a lot of delusion in this field around pay, so it's important to be well informed. As a field we are still very well paid compared to most other jobs especially considering our safe working conditions and lack of needed credentials and education. Compared to most of the work on this planet, it's still a goldmine.
That doesn't match my experience on both ends of the hiring table. And forgive me if forwarding the BLS statistics to candidates doesn't get them to accept offers, because I know it wouldn't help me when I can get paid a lot more elsewhere.
> That doesn't match my experience on both ends of the hiring table.
Congratulations, your experience is limited. The BLS stats represent the actual US salary data, not just your limited experience. If you want to make a claim about salaries in the US then look at data across the US and not just whatever is true within your limited bubble.
> And forgive me if forwarding the BLS statistics to candidates doesn't get them to accept offers
Come on! Everyone posting on HN makes $500K a year, has a vacation home in Tahoe and drives a brand new Ferrari. They think "L6 at Google in the Bay Area" is the median job level for the entire software industry.
Is 7 figures even realistic? I was under the impression that even a senior FAANG job was still just under $300k typically. Does anyone outside of a CTO/CIO make $1mil+?
I know half a dozen of Senior Staff at FAANGs and they make I would say around 600k - 700k, more of course if stocks appreciate but that's hard to control.
Consistent total compensation offers of $1mil+ is probably reserved for roles above these, sometimes called Principal or Distinguished. I would say the rate of having one of these in an org are like 1 for ~150 engineers.
If you sample only from the 10,000 or so current Olympic athletes, you will draw similarly incorrect conclusions about the 8 billion planetary general population's athletic ability.
That's either a fun or terrible exercise depending on who is administering and how. However, if you said it's a Ruby role and the candidate is good enough to be picky, you may scare them off when they think your description of the role was a lie.
I agree. I've had this happen often enough on the job that it's not a totally made up example. And usually you'll be one of two or three people in the whole company who is able and willing.
Debugging old DSL vendor specific languages or code so old using, frameworks and standards long out of fashion and support, that they are half way to being a different language.
Adding support for some back ported features or patching security holes in an old client or legacy stacks.
Or at a big company we had some escrow code from a much smaller partner that we ended up becoming responsible for.
Often getting the environment setup for proper debugging is more work than anything.
Most of my development experience is in C/C++ and Python.
Know what I'd do if the interviewer asked me to debug PHP? Pretty much return the question:
"I've never used PHP. Are there logging macros/functions defined somewhere? Where do I see the output? syslog maybe? Is there a debugger of some sort I can use? How do I run each `piece` of code in isolation?"
(I am assuming the job listing did not explicitly mention PHP experience. If it did, both myself and the recruiter would absolutely deserve to fail me for this interview).
I interviewed for a Scala role one time despite never having written it professionally. I suppose it's obscure enough that they couldn't afford to be too picky.
It was a pair programming exercise and so with some help from the interviewer and the IDE I was able to fumble through to a working result. I agree it was fun and educational.
being able to do that without significant time constraints isn't that bad imho, but inside of an interview?! That's borderline laughable, for me, as you're only going to be filtering out people based on whether they have any previous PHP exposure or not.
The more people online complain about coding interviews, the more confident I am that they are the absolute best way to filter candidates for a software development job. Across the industry there are way too many talkers/pretenders/meeting schedulers and not enough people who can roll up their sleeves, jump into the code and actually get stuff done. And this problem becomes worse at higher levels. You can bitch about it all you want, but you aren't owed that cushy $500K/yr FAANG job. If you can't get yourself to brush up on basic programming and write some for loops then companies will simply move on to someone who will.
I am good at passing these interviews and am at a faang (will be moving to another one this month). These interviews are useless and provide a false signal on problem solving skills and people's abilities to learn things. The interviews specifically don't test whether or not somebody can roll up their sleeves and jump into code, because if it did why do I as a new hire have to explain so much about software engineering and debugging practices to people who have been here so long?
If you've actually worked at a large company, you'd know that 90% of the real work is done by like 5% of people (maybe even less). If the interviews worked this ratio would be so much better.
Yea, I happen to hate coding challenges, but not because they're hard--because they bias towards "people who have time to do coding challenges". That said you're absolutely right about how many phonies are in the industry and coding tests are probably the best we have to weed them out.
The problem I see in a lot of cases is putting too much emphasis on solving the problem in the interview, instead of working through it. Many interviewers claim that they care more about the latter, but in practice failing at the former ends up disqualifying you.
Agreed. And a lot of people will say that you can just memorize a bunch of leetcode problems, but I've always created custom coding problems when I've acted as an interviewer so good luck finding the solutions online.
And frankly, if they're able to adapt a random leetcode problem to be able to solve the coding test that I give them, then that's exactly the kind of adaptability that I'm looking for in a prospective software engineer.
Typical table stakes prep work for a high paying job with a lot of competition.
I’m serious. It’s not even guaranteed you’ll get the job if you do the above but everyone who passed the big tech interviews will acknowledge they fucking studied for it. What do expect here?
The ease with which interviews allow the unscrupulous to inflate their abilities has been well-known for long enough for it to be metagamed and exploited for profit and now the effects have become abundantly apparent. Imposters institutionally infested in this manner across industries have reached a critical threshold where their fundamental lack of competency can no longer remain hidden.
If you do the above prep work and pass the multiple interview loops you’re pretty smart honestly. The vast majority couldn’t do it even with all the study possible and even the smartest couldn't do it without any study at all. The bar is pretty high.
Even if we buy that leetcode is good in general, what about for the increasingly more dominant variant, the ML/AI engineer?
How can we even ban them using LLMs in their "ML-leetcode" problem when the problem is on LLM tokenization or something?
Like, if I lose a candidate because they're experts at using cursor + claude to do their coding (i.e. they solve it by writing 3 prompts and filling in 2 errors which are easily spotted in a total of 10 minutes), despite them having NeurIPS caliber publications and open source code, did I filter out a fake grifter, or did I filter a top tier candidate?
I just don't buy that leetcode style problems make any sense for anyone whose doing anything involving LLMs, and like it or not, increasingly large amounts of the cushy 500K/yr FAANG+ jobs will involve LLMs going forward.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Literally every day this week.
Documentation? The code was last updated in June, the documentation was last updated in January - of 2022.
The team? The half that didn't get laid off two years ago all quit for greener pastures. Sure, there's a team that owns this project - along with three others, all of which they've contributed maybe a dozen lines to over the past year, and 11 of those were dependency updates.
I have no issue with doing this on a job, but i have issue with it in interviews. Not everyone is able to be comfortable in a strange environment with strange people. Give me a damn take-home test and stop talking amongst yourselves while i'm at the whiteboard.
I had a very bad experience at a company that did not conduct any coding tests; there were just 2 or 3 interviews, and that was it.
The issues began when we started working together in a Data Engineering team. Most of our stack was based on Hadoop, Kubernetes, Ruby, Python, and several other technologies that required a basic understanding of cloud computing.
However, since we had such a wide range of work experiences and backgrounds, I often found myself not doing the work I was hired for but instead covering for others. I ended up doing a lot of "glue work" to compensate for the fact that some colleagues couldn’t even handle basic tasks, such as Python CLI packaging using Docker or inspecting jobs in Hadoop. After some time, I decided to leave—not because I thought I was special, but because I was fed up with spending 80% of my time providing support for people who were hired at the same level or even higher than me.
I recognize that the company had very poor hiring practices, but some form of basic technical testing is necessary.
When you say they "couldn't even handle basic tasks" do you mean they were completely unfit to complete their assigned tasks, or that they were getting hung up on simple tasks, while otherwise being independent and capable? The other key question is whether you had to explain the same things to the same people again and again. Being ignorant is excusable, but being unwilling to learn isn't.
> they were completely unfit to complete their assigned tasks, or that they were getting hung up on simple tasks
They were unfit to complete the tasks. But again was not their fault, I think was the hiring that failed.
One concrete example was the fact that our pipelines was quite straightforward for data engineers: packaged Ruby and Python CLIs that runs commands. The runtime was k8s. One of the biggest issues were when something broke in production, none of the people couldn’t go to the container, check the logs and understand the failure.
There’s one situation where I think makes sense to hiring without a test: if the company has the resources and money to provide levelling training for all new hires with no exceptions. I do not know how practical it is.
Code challenges are great if well made, and horrible if not. They have to be short enough not to be unpaid labor (few will go through the trouble of paying their victims to work on interview tasks, and if I have an existing job I don't want to plow even 8 hours into getting any new job).
But having a simple coding task does some great things in that it will make the interviewee have to fix problems. They need to ask when unsure. You'll know who stops and asks and who plows through and solves the wrong problem. You'll know who throws their hands up when instructions are unclear and just says "I can't do this" and gives up. It's one of those extremely valuable and very effective filters that you just shouldn't be without.
The actual assignment can be however simple, but if they manage to install the tools, clone your repo, read the instructions, build the thing etc. then you already filtered out half the applicants or more. And that's without even starting to code.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
The fun part of take-home assignments is that there's a moral hazard incentive to spend extra amounts of time on an assignment even if they say "only spend X amount of hours on it," even lying amount the amount of time spent if the interviewing company asks.
More than once I have completed a take-home assignment with emphasis on sticking in the timeframe they recommend but still solving the problem adequately, and then I get chewed out in the followup interview "why didn't you implement X?"
Relatedly, a massively open-ended take-home assignment is a bad take-home assignment since it favors those who have more free time to be thorough. (shoutout to the take-home data science assignments which ask for the final deliverable to be a PowerPoint presentation)
> Hiring processes should focus on problem-solving, collaboration, and growth in relevant areas. Unrealistic expectations don’t attract the best talent – they just exhaust and discourage it. If companies want adaptable developers, they should focus on the long-term ability to learn, not how fast someone can tackle an arbitrary test. Dropping these absurd assignments and focusing on what really counts could foster a better, more inclusive tech culture.
Sure, how? There are no perfect processes, only drawbacks that you can live with.
I also find the idea that interviewing is somehow exploitative of the interviewee ("feel like working unpaid shifts for a job they don’t even have yet") to be somewhat bewildering. It's not like your coding challenge is useful work for company that you are doing for free, in most cases interviewing is also a huge time sink for the company as well. Not every interaction should be quid pro quo...
I think the reality is that it's too costly to put together a process that filters better. This has been deemed "good enough" and they can fire you if the filter didn't work properly.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
> This is like asking a Ruby developer to debug PHP as a test of flexibility
> If the job requires specific tech skills, test those skills
Sounds like someone failed a coding challenge.
In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with. Telling them to get rid of it without attempting to understand what problem they're trying to solve and without offering alternatives, is, to say the least, not a productive use of anyone's time.
I find in software development we try to reinvent the wheel all too often instead of borrowing practices from other professions. Let's have a look at what civil engineers have to go through to get hired at half the salary of a software dev, and let's incorporate that into our practice. That will make everyone a happy trooper !
> In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with.
Unlikely. It's usually what the most senior person either read about or experienced in the past. In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
They just did it like everyone else because they believe exactly what you do -- that someone somewhere created the process with intention.
I feel like we've so lost the plot with tech hiring that we settled on what appears at best to be a local maxima.
Folks on this forum mostly won't remember but about 20 years ago the in-vogue way of interviewing software engineers was to ask "puzzle questions" (e.g. "I have 100 ball bearings and two are a different weight than the rest....") and "lateral thinking" questions ("why are manhole covers round?") because that's what they did at Microsoft and everyone copied Microsoft.
I'm told this is still the common style of interview for mechanical engineers, which says something about what it's like to work in that industry too.
FB interviewer had "two hard Leetcode problems within 35min", where 25min was background and 1st leetcode problem was "changed" (adding ambiguity to the point of be without solution - later could not find any full solution, for which I stated factors and disclaimers to interviewer).
Agree, such or similar unrealistic interviews are loss for both sides, plus they are plain STUPID. I've performed ~400 interviews that most included coding in my career with consistency and standardized report (can compare apple to apple).
I was ready for this to be another complaining post about having to write code during an interview, but found myself mostly agreeing with it.
Having to spend time to do some work before an interview to prep (like a presentation about your experience or something) seems reasonable, but a company should be able to test technical skills adequately in the 5+ hours they spend interviewing candidates.
I feel like the take home assignment trend is part of the backlash against doing coding problems live in interviews and I agree with the article that it's just worse. It takes more of your time and also opens things up to candidates cheating even more than live coding over zoom does.
Just stick with the live coding and ask good questions.
Had the weirdest interview of my life on Tuesday, more on that at the end.
Relevant part: the interviewer/CEO's philosophy is not to code challenge because no one properly vetted for "problem-solving, collaboration, and growth in relevant areas" (tfa) would ever accept a job they were going to fail at. Will always stick with me.
Now, here's the weirdest interview of my life: An acquaintance who runs a software business interviewed me by bringing me along to an NBA game he has season tickets for. The interview in the dining area before the game was normal, the game included some breaks for more interview discussion, but the home team had the best shooting night of their entire season so the crowd was very rowdy and we each had 2 beers.
The part I always feel like is missing from the coding interview discussion is:
"Are you testing the candidate on what the job will actually require, and ONLY that?"
With an onsite whiteboarding challenge, you're most likely testing if they're really good at thinking and talking in front of a crowd when extremely nervous, not writing code.
Same with a live coding challenge over zoom (CoderPad style) - this tends to be a test of nerves more than coding ability. But, if you're planning on a lot of pair programming, and want them to be able to dive into that on day one, maybe it's the right way to interview.
I find take-homes often the most accurate way to interview, as it most closely reflects the reality of most coding jobs. You're on your own, you don't have to deal with the extremely odd (and for some of us terrifying) feeling of someone watching you type. Hopefully, the take home questions are carefully designed to reflect what the job requirements will be. As almost every other comment has pointed out, debugging an ancient code base without documentation may be a perfect question, or it may be completely irrelevant and unnecessarily eliminate candidates.
But please, make the interview match the requirements you're actually hiring for, otherwise, everyone's time is being wasted.
> Are you testing the candidate on what the job will actually require, and ONLY that?
As a hiring manager, I strive to evaluate for potential and adaptability, not a highly specific list of skills. I think many candidates prefer that, too, that they will be allowed and encouraged to learn how to do this job. I do not want to hire people that don’t want to learn or can’t do things they haven’t done before. And I may or may not even know exactly what the job will require. Do not assume your interviewer can tell you exactly what will happen once hired — they can’t! One thing I do know, for sure, is that the job will require communication with others, adaptability to changing requirements, and a willingness to learn the company’s workflow. I need to make sure those boxes are checked, sometimes more than I need to see how well someone knows jQuery or CUDA or sorting algorithms.
> build a mini-app from scratch in just a few hours
It depends on what kind of functionality we're talking about, but this kind of task is exactly what people at my current startup have been assigned at times. It is absolutely possible to build a CRUD web app with reactive UI using modern tools in a few hours.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
Again it depends on what the debugging task is. At every startup I've worked at, it's expected that an engineer is able to jump into a task that they know very little about. Granted it becomes less reasonable the more niche the task, but PHP and Ruby are not particularly far apart in skillsets in the grand scheme of things. I would expect any web engineer to be able to do this.
> Hiring processes should focus on problem-solving, collaboration, and growth in relevant areas
I agree with this. And, hiring should also focus on technical ability which does include working through difficult and unknown problems by oneself.
> is absolutely possible to build a CRUD web app with reactive UI using modern tools in a few hours.
Yes, but it also leads to tons of bikeshedding. "Should I implement a linter? Should I write unit tests? Integration tests? Should I mock HTTP calls or use a mock server? How should I deploy things? How do I structure my CI?" All these choices are extremely subjective and context dependent and then somebody's gonna pass on you because they don't like that you used Cucumber.
Setting up a project is not something you do often, and if you do, you probably have templates and some team standards.
It's much better to give somebody a project structure and ask them to implement some basic task in it.
> build a mini-app from scratch in just a few hours
But how often do devs normally set up a project from scratch? We have like 3 new apps in a few years where I work now in addition to the long running ones. So on average one in like hundred devs here have been part of creating something from scratch.
Sure, when you know what you're doing it's quick. But the first time can be slow, no matter how good you are. So for a coding task that should take a few hours, you might have spent all the time just getting up and running, and then you actually start the task as over time.
Remember that in other fields like medicine, finance, academia, and law, getting in involves 5+ years of hoop jumping and commitment signaling that have nothing to do with the final job.
I have a couple family members who are doctors. Medical school and residency were nightmares, but after that they are golden. One of them works one week a month and makes 400k. Others have pretty cushy 9-5s and are taking home ludicrous salaries. They are also pretty recession proof and their salaries are fairly immune to economic pressures. I don't think mass doctor layoffs happened in 2008 or 2022.
With software you never have to stop proving yourself, and your skillset is always a few years away from being outdated. A doctor with 20 years of experience would be welcomed anywhere, but an engineer with 20 years experience is viewed with trepidation. The next "big thing" could roll out at anytime and suddenly crypto engineers are getting 800k job offers so everyone furiously tries to learn crypto stuff. A few years later that all dries up and now you have 100k engineers who are out of work and learned tech that no one cares about anymore. All the LLM engineers might be in the same position next year.
Genuinely curious, knowing nothing about this field: does, for example a neurosurgeon with 15 year of experience, when looking for a new place to work, have to pass surgery "challenges", like doing a lumbar puncture in 15 min on a fake body to prove his experience?
No, but the medical profession has a ridiculous amount of gatekeeping, so you can't become a neurosurgeon with 15 years of experience in the first place (there are maybe a few thousand worldwide). On the other hand anyone with a computer and a curious mind is a software engineer.
Let me answer your unknowable question with another one:
Which doctor would you prefer do surgery on your brain - the one that jumped through all the hoops and went to 15yrs of med school - or the one that learned brain surgery through a YouTube tutorial?
I think you know the answer.
The problem is not gatekeeping in general it’s the detail and nuance of how gates are being kept and how to find the right amount of gatekeeping.
> Which doctor would you prefer do surgery on your brain - the one that jumped through all the hoops and went to 15yrs of med school - or the one that learned brain surgery through a YouTube tutorial?
My answer is that I don't have access to what I'd really like to base that judgment upon: their success rate, adjusted for the difficulty of the procedure.
Simpler, less-risky procedures would therefore have a lower skill/experience bar.
Basically, I'd want someone with decades of experiences to remove a tumor deep inside my brain, but I'd be much more willing to use someone who learned via YouTube this time last year if I'd suffered a TBI and was experiencing swelling and intra-cranial pressure.
No, but they'll ask for (and actually follow up on) references from their fellow surgeons, and check that they've passed their board exams, and that they're licensed in the state, and that they haven't had huge malpractice claims, and otherwise verify that the person is, in fact, actually a neurosurgeon who does neurosurgeon things in a hospital and is acceptably good at it.
It's not a perfect system, but it's a lot harder to fake being an experienced doctor than it is an experienced developer.
No, because the fact that a neurosurgeon with 15 years of experience is not in jail means they know what they're doing. Not the case for devs unfortunately.
Dunno, all my "normal" engineering job interviews just revolved around my previous work, going through big picture design problems, and specific things related to the job.
Didn't have to solve PDE's on the whiteboard, or regurgitate integral transformations.
True, but someone with 10+ years in those fields doesn't face an interview asking them to prescribe treatment for a cold or explain the difference between civil and criminal law.
It’s not insane for companies to filter candidates. More people want to be highly paid software engineers than positions available.
The insane thing would be to expect someone to take you at your word for 150k*, when hiring managers know that 50-70% of people with your resume fail to write a nested for loop.
I feel like the real problem is that most people are poor evaluators of talent. The pervasive bad interview processes in industry reflect this truth (as well as cowardice that allows people to hide behind process to avoid accountability as well as accusations of bias). The only way to escape this trap is to find the right connections or start your own org. Otherwise you will spend your entire career suffering the consequences of broken interview processes (both in terms of your experience getting hired as well as being forced to work with bad hires who game the system).
We recently went through a hiring process using a recruiter. Our assumption was that the people they vetted were "legit" and we didn't need to do any additional work to verify that they had the skills they said they had. If the recruiter wastes our time it's a surefire way for them to never get business again.
Anyway, in lieu of a coding interview, I opened up a couple of tickets (one a long term change request, another a real life problem we encountered just the day before). I provided them logs and answered questions and watched the interviewees move through both scenarios.
We capped the time at an hour. The focus was more on the interaction between them and myself—it actually used the "real stuff" they'd encounter.
I was amazed at the diversity of the responses and how differently people approached a problem.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Pretty much all the time. I also found that debugging things without asking for too much help was a great way to learn the systems involved, which eventually made me a better and more useful programmer, with more domain knowledge.
Granted not everyone has the luxury to spend time doing this, but I don't think it's that rare. In many cases it's even what companies want you to do, when that documentation or help is simply unavailable.
Non paid solo-coding assessments should disappear. Many disrespectful companies send it automatically to all applicants to don't even look at them after.
Every time I get those I request to do a live coding assessment, if they are not willing to lose even 1h of their time they were not serious about you.
If you decide your process really requires a solo assignment you can offer a gift card to prove you appreciate the candidate (and don't be stingy, you are already saving a lot of money by not having 1 of your engineers interview)
The take-home challenges at my last jobs have been close enough to the work to be useful, and much more relevant than anything else I've seen in any interview.
Frame shift: coding challenges are work. (I don’t care that the assignment is useless for the company, that’s not my problem!) The issue for me is that it is unpaid.
If you really want me as a developer, pay me. As an industry, we might even set a standard rate for these homework assignments. I suggest a rate which is enough for me to say it wasn’t a waste of my time.
As an anecdote, I was interviewing at a startup that sent me one of these homework assignments. I told them I would do it for $1000 now, or whenever I got around to it otherwise. I never got around to it.
At most places I know, a coding challenge is not "the way" to get accepted, it is a way to get rejected.
The hiring manager might hd really liked a candidate and conclude that a coding challenge just shows that the new hire would need extra time to ramp up.
Yet, ignoring money, you could excel in the coding part and the hiring manager might not like you because you lack "X" (e.g. "communication skills", "team culture", etc), which in reality is a way to put they just didn't like you or there were not really looking for someone to deliver, example, they were looking for someone to play politics or that might benefit the hiring manager more in the long term.
Not always the more technically capable candidate gets the job.
Whiteboard coding interviews are underrated. Sure, they suck to learn and master, but once you gain the experience it's a passport to a wide range of top-shelf jobs.
Can people game them? Are they flawed? Absolutely. But that's true of literally any interview practice.
The idea behind whiteboard interviews is if they're challenging enough, the only people who can fully exploit them are either very bright, very motivated, or some combination of both. That's a better signal than asking someone to lie about their "passion," which any slob can do.
I apologise for the shameless plug but I’m working on something to solve this.[1] The problem is that most hiring can be classified into 2 types - the company needs candidates that can work with their tech stack OR they want candidates who are good, regardless of tech stack. What we do today is conduct interviews that make no sense and that have vague signals when we could be assessing them based on real-world skills. Candidates shouldn’t have to practice interviewing in any case!
If you are a hiring manager and would like to experiment with something like this, please get in touch!
Eh, coding challenges in of themselves are fair enough. What companies really need to understand is that:
1. Said challenges should reflect the work done on a day to day basis, not something completely disconnected from the company as a whole
Google's interview challenges should be very different from say, a local web development agency's interview challenges. I've seen too many companies where either the challenge was way too difficult/complex for the job on offer (build a complex modern app from scratch for a junior/graduate level role, or solve a bunch of leetcode tests and HackerRank puzzles for a WordPress agency job), or where it was far too simple (I remember at least one software engineering interview for a React focused role where the challenge could be solved without writing a single line of JavaScript).
2. The requirements and environment should match a realistic working day
When was the last time you had no access to the internet, no access to an offline IDE, no access to premade software, no one to help, a boss standing screaming at you over your shoulder and a camera tracking your every eye movement in case you're 'cheating'?
Probably never right? Unless your company environment is a dystopian hellhole, this is probably not your typical set of working conditions. Don't expect interviewees to work under them, they're not kids in school doing an exam.
3. And that said projects should be somewhat realistic in terms of scope
Most people aren't coding a new website/app from the ground up every day/week. Expecting someone to create that in a few hours feels incredibly unrealistic, and an incredibly poor method of judging someone's skills in the role.
I've turned down a number of jobs in the last few years because of unrealistic interview requirements.
One asked for a 20+ page personality assessment (perhaps completing that _was_ the test?). The other asked for an interesting technical challenge which I was happy to do -- for fun -- and then followed up with a request to write a basic web service which was... unexpectedly simplistic for the level of role I was applying for.
I pointed out a number of my public, solely maintained repos on Github illustrating exactly the same sort of thing they were looking for and turned down the followup assignment, but we decided it wasn't a good mutual fit.
* Some companies design take-home assignments that mirror challenges their team has spent years refining, setting unrealistic expectations for candidates to match or exceed that work. Talent is sometimes dismissed over minor differences, like testing styles or even coverage percent.
* Working with people is about compromise. When passionate people collaborate on a creative endeavor, there's always some back-and-forth of ideas. But in these assignments, often only the unilateral opinion of the evaluator matters. I guess being rejected from a monoculture early in the process may actually be a blessing in disguise.
* Many companies fail to provide specific, actionable feedback after candidates spend hours on assignments.
I don't mind take-home assignments, but I avoid investing excessive time in them, as it’s not sustainable and often has low ROI.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Yesterday! Working on a large project means there’s always some issue with a dark corner no one has looked at recently and because there’s no team to support, I get to go and bug hunt.
Many of us grew up when that was the norm, n00bs weren't given features. Coding jobs were like apprenticeships. You were mentored and free to explore. You learned what did and didn't work and how to leave the codebase a better, more maintainable place.
Today, privilege means starting from no real job experience to writing enshitified_prod_tokenizer() and laughing at the poor sap having to fix it in a few years.
> What companies often ignore is the extra time candidates invest beyond the “suggested time” for these tests.
This is a feature not a bug. Companies are testing if you can focus and complete a hard uncomfortable challenging task, because at your job you’re expected to do things you don’t want to do, but will be rewarded for doing.
Companies are backhandedly selecting for those who can invest the most time beyond the suggested time (read: single, childless people in their early 20s who live at home) and those who are willing to work insane hours. It's a pledge of fealty before you are graced with an interview. 15 pieces of flair is the minimum, but it's up to you whether or not you want to just do the bare minimum.
Requiring a huge time investment like that will filter out a lot of the non-desperate folks…probably exactly the folks they don’t want to filter out!
Also do they really want to set the tone that “I expect you to intuit what I want and I won’t tell you directly”? Sounds like an awful place to work, right out of the gate.
> probably exactly the folks they don’t want to filter out!
Maybe? non-desperate folks might waste the company's time. Usually companies can only give out one offer at a time. If the non-desperate person takes a while to accept, someone else in their pipeline may get an offer and flake.
We had a candidate agree to a start date and then cancel 2 weeks before because they decided to stay at their job.
> “I expect you to intuit what I want and I won’t tell you directly”?
They do tell you directly? "Do this hard uncomfortable task, where the task is study leetcode, completing a coding project."
That's a perfectly fair and reasonable tone to set because that's how approximately 99% of non-junior jobs work. My boss says he has a problem he'd like me to solve. He might not have all the details so it's on me to investigate the options, identify the gotchas, and clarify the ambiguities.
I never deliberately snuck ambiguities into coding challenges. I've used it in oral sessions for senior and above devs though. "I'm a product manager who wants to add a new feature. I don't really know what's involved but I want you to implement it. Feel free to ask me a million questions and we'll talk it through!"
> Also do they really want to set the tone that “I expect you to intuit what I want and I won’t tell you directly”? Sounds like an awful place to work, right out of the gate.
I have only once had this be a problem and in a way, it was my fault for not noting the ambiguity in the submission. Every other time I have dropped a comment saying "you could have intended A, but also could have intended B so this is when I go back to product and request more information."
i think we need to come to terms with the reality of coding challenges in the interview process. i know i hate them personally, and dread having to interview again because i'll need to open leet code and remember how to do stupid shit like DFS on a graph, or manipulating linked lists. at the same time, a job opening for a SWE is opening soon in our company and we'll have to somehow filter people, and the job market is such that we'll get MANY applicants, most of them probably wrong for the job. i will probably end up giving them coding challenges (not necessarily leetcode, but some coding challenge for sure), because i need a way to grasp their problem solving and coding skills. i don't know a better way to do it in a condensed time frame of a 1 hour zoom call.
I'm fine with tests, but only if companies pay a standard fee (say, $100) for a dev's time. If a company doesn't respect your time during the interview process, it probably won't while you're on the job.
Agree, IMO even interviews should be compensated.
Most software companies want to schedule 3-4 interviews each 2 hrs long so they should minimally compensate 1 day pay.
If we assume 15 days off that leaves roughly 245 workdays per year and with a salary of $200k that would be close to $800.
A fair amount of compensation to spend half a weekend on a project would be $400.
If someone gives me a take-home assignment that I spend 4 hours on, there is no commitment of time on their side. They may decide to not even look at the results. They may spend five minutes on it.
That's exactly my problem with take-home code challenges. If I go to an interview, and they waste my time, they have to waste their own. But with a take-home assignment, they can waste my time without wasting their own. That may lead them to be more wasteful of my time.
I've recently had a couple technical interviews where they were structured much more like a pair programming session. Those were a great experience on my end, and for the interview I expect it gives a much more realistic view of what it would be like to work with someone.
I agreed with most of the things in the article. However, the debugging test is very reasonable provided they give you a heads up on the environment/language you're meant to debug. Depending on the language, setting up a debugger can be a test in it of itself.
Debugging (and setting up the debugger) is a core skill every developer should have.
This happens when companies have too many applicants. It isn't the best way to evaluate candidates, but it is a cheap and easy way to do it and resumes aren't enough.
It's common for candidates who have an internal referral to skip all these steps, because the referral establishes a base line of credibility.
we administer a coding challenge designed to ensure people can do some basic coding - we don't even have them compile it. But we have caught people who start talking about using a B tree to solve fizzbuzz and we know they're not a fit. It's useful as an initial screener.
> Dropping these absurd assignments and focusing on what really counts ...
These assignments are so much closer to a simulation of doing the actual job than the industry standard LC interview. The only thing closer is work trial which has the significant problem of requiring candidates to not currently have a job.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Literally my first internship. Had to migrate a small system from C to (modern) C++ in the span of 6 months. Most files (and I'm going to say in the ballpark of 85%) had last been touched in 2003.
I wholeheartedly disagree with this take. there's nuance obviously and takehome coding challenges can be done horribly but when done right and assessed properly, they are a wonderful tool for hiring.
Unfortunately, you often have to ask the interviewer (which is tough when it's a coding challenge and you have no way to ask for clarification on the rules). I know interviewers who will say "You didn't cover edge case A, B, and C, your solution is incomplete!" and other interviewers who will say "You are overthinking this, I'm just looking for an MVP."
Pretty common artifact that take-home challenges are being more widespread now that the labor pool is oversupplied with tech talent looking for jobs. Companies need new ways to filter for candidates that are willing to do whatever it takes and work the hardest. The easiest way to do so is give them a "three hour" take-home assignment and see if they are willing to do it.
Doesn't this approach self-select for people who
- value their time less,
- don't have other things to do (family, hobbies, side-projects),
- unemployed people
It's one thing where companies are applying this to 1000 applicants for an entry level position, but they also use this tactic when they reach out to you and tell you how excited they are about your background and resume.
In reality they didn't evaluate your background or resume and are just cold-calling you to fill some portion of their "funnel", then ignore all their excitement and reasons they reached out to you and have you jump through random hoops.
Aside from all the comments saying that coding challenges aren’t as unrealistic as you think, there’s also the fact that there seems to be an oversupply of good enough coders for some roles so companies need to filter beyond that.
Yeah inclusive environments etc is a nice to have but if you can get a top level coder that can solve problems quickly and not crumble under time pressure, even better.
Why settle for good enough if you can get really good instead?
I’ll act as a counter-balance on this topic. I have conducted hundreds of technical interviews across varying levels of experience - from interns and juniors all the way to PhD grads and those with lots of years under their belt. I have extensive hands-on experience in multiple fields and passable knowledge in others. If a candidate has worked on something in the past, a good position for me to be in is to know more about what they’ve done than they do. A passable position is one where they can walk me through what they’ve done and I pick up as I go along, piecing it together in my head with what I already know. I very rarely have pre-canned questions, instead I opt to come up with examples on the fly based on how the conversation (and it is a conversation) is unfolding. If the candidate is gold, I will usually have a first offer prepared for the candidate before the interview is over, and I do not cheap out on the offers. Candidates have historically reached out to express they enjoyed the interview even if an offer was not presented. I believe in respecting the candidate, respecting that people can be nervous and shy one day and social the next, that people interview differently, and that more interview rounds does not necessarily yield a better result. Prerequisite for this is you have to enjoy talking to people, put your bullshit ego aside, show your hand where you think it will help the candidate, and know what you’re talking about and what you don’t know. People can learn, people can adapt. Every so often, I accept candidates that would otherwise be rejected on the understanding that no “system” is ever perfect, certainly not one that involves people. It saddens me to see coding challenge, especially automated ones, because it shows laziness on part of the interviewer. However, it is a skill that must be attained and practiced like any other. And no, I haven’t used the return key since the 8th grade. AMA.
I can't claim to be always be good at it, but this is what I strive for as well. I like talking to a candidate about something they are experienced or even expert at, and asking questions until I find the boundary of their ability. It can be very illuminating what happens when they hit it -- some get defensive, some get enthusiastic.
Defensiveness might just mean they're being interviewed and I failed to adequately put them at ease. But usually when we're neck-deep in some specific topic, both of us will kind of forget that it's an interview -- especially since the topic is more in their area of strength than mine.
Otherwise, defensiveness often means they never actually wanted to understand the problem they were working on, they just wanted to use it to get a degree or ship something by throwing it over the wall and forgetting about it. Even someone who is heartily sick of their PhD topic (as in, everyone who is over 1/3 of the way to getting one) will be relieved to discuss the ideas behind it, the reasons why it caught their interest in the first place, when they don't have to do the work of coming up with rigid results and writing them up.
Enthusiasm is usually a good sign, though even there I have to watch out for excessive enthusiasm where they care more about the problem itself than the benefits of solving the problem, and are likely to waste resources in unnecessary pursuits of perfection.
Oh, and finding the limits of someone's knowledge doesn't require a genius, which is fortunate since I am decidedly not one. 3-year olds can do it just by endlessly asking "why?" You'll probably need to be a little more sophisticated than that, which is good since you'll be able to evaluate their ability to explain things to you in the process.
>When was the last time you had to debug an ancient codebase without documentation or help from a team?
Uh.. just about every company i've worked at in the last 20 years has had little to no documentation and the guy who last touched the code left already.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
I've had to do this at some level in every job.
Somebody left. The code was left to rot, whether it was a separate codebase or it was an entirely separate project. I got assigned to fix it. Sure, I had a team, but nobody knew more than I did, so it was just other devs who might have another perspective.
Collaboration and support might be standard, but mostly in the form of other smart people, not always documentation, up to date software, or people who know anything about the code.
I personally prefer these kinds of challenges as they mean I don't need to maintain two skillsets.
I had one coding challenge a month ago that said: this should take you no longer than 2 hours. It didn't have guard rails on it so you could technically spend more time on it. It ended up taking me 2 days on and off because it was basically build a front and and a backend. And then later I realized the instructions for submission were totally broken too: DONT make a PR - later in the instructions: make a PR.
While we're at it: enough with the brain teasers and weird analogies. These, too, have nothing to do with actual day-to-day work and are really just a way to exercise power over candidates.
This is a friendly reminder that you are welcome to withdraw a job application as soon as you are offered a coding challenge. If this is an exchange with a recruiter over email or linkedin messages, you can simply reply with something like this:
"Thank you for sending that along. I would like to withdraw my application for this position."
I was asked to code a sudoku solver from scratch during an hour long tech screen. I got maybe 60% there. A few days later they called me up and said they liked my work but wanted someone faster. At the time I was bummed, but a decade later I’m glad I didn’t get that job.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Right the fuck now. That's what happen when you get saddled with whatever monstrosity the last coder(s) who quit commited or had to maintain themselves.
And then you get the joys of trying to frankenstein some modern js "component" developed like only React exists on some vanilla / jquery front. Also a personal rant for those modern js script kiddies: forms action attributes exist and work.
I once ended a job interview at a company because of their coding challenges at the end of a 2-3 hour interview. It was a job for a PHP developer working in a DDD environment. They were asking people to do algorithms, the last one just took the cake because it was completely unnecessary.
The question made sense: how do you add to very large numbers together? I assumed the key knowledge they wanted was that the numbers had to be represented as strings and you should use bc_math to compute them. So I answered that, and they said, "Yeah, great. Now write a function to do that." I sat there for a few moments thinking about it and realized it was just nuts. So I asked if I could just end the interview process. They were clearly shocked, I doubt many people end an interview process on the last question. I said that this wasn't realistic and that if I did that for real it would get code-reviewed to hell. One of them kept saying but it's to see how you would do it if bc_math wasn't installed. I simply stated "I would have them install it" to which one of the interviewers instinctively nodded since that's the real-world answer.
It seemed like it would have been a good place to work up to that point. I didn't like my current job at the time, it paid considerably more, had more public holidays due to weird German reasons, etc.
Ugh, I got that question in an interview. Only they wanted me to write up a basic arbitrary precision math library, in C, on paper. (I may have chosen to do it on paper.) It was harder than I expected. I didn't do that well and didn't manage to finish in the time allotted -- halfway through, I switched from encoding things as ASCII numerals to base-256 binary (or the reverse?) -- but it was enough that the interviewer said it was clear I understood what was required. I got the job.
I thought it was a hard but fair question. I still feel bad about not doing well. It wasn't all that close to what I'd be doing in the position, but it wasn't unrelated. Perhaps if it were a PHP role I'd be more bothered? But even there, it seems like one way to check if someone understands what's going on under the hood, so as to not naively do expensive stuff without realizing it. The fact that one should not write their own library, and that if you tried you should get code reviewed to hell, isn't relevant if that's the purpose.
I don't mind the coding challenges necessarily, but more how they're implemented. I find them to be far too broad relative to the day-to-day, and it only showcases how quickly someone can write unthoughtful code.
For example, it's not uncommon for a challenge in finance to be "build an exchange matching engine that takes orders, modifications and cancels from different threads in two hours." That's totally doable if you just fly through whatever comes to the top of your head, and maybe have an hour left over to test, debug and modify it. But that project (an exchange matching engine) is something that the CME has 10 full time programmers working all the time on. A more realistic 2 hour task is something like "look at edge cases for the order cancel type, and figure out a way to handle malformed inputs." Then you can actually write production quality code for them to judge.
As interviewees, we need to stop entertaining this nonsense. A company tried to give me a "homework" assignment and I just told them I'm not doing it and moved on.
My employer's coding challenge uses replit and takes 30 minutes in which we're watching them do it (and interacting, asking them questions or prompting them with hints). It's not a hard challenge at all--one candidate who'd been grinding leetcode did it twice in the 30 minutes, with different solutions. We do the same with a system design question.
Most finish it, some don't and still get hired. It's a workalong in which we see how they approach the problem, how they discuss it with me, and what they do if they get stuck. We tell them to go ahead and google things, just tell us what they're looking for.
Over 10 years, we've got a pretty good record of hires that are smart, professional, and effective. And without specifically trying to hire for diversity, we've had a surprisingly diverse range of hires. We don't care about "culture fit", it's more like "work fit".
So I agree that the sort of code challenge where you send them away and see what they come up with is both unfair and a poor indicator. The real hiring test in an interview should be "can I work alongside this person?"
Nope. I gave dozens of coding challenges at my last job. It was as directly job relevant, and as low-stress, as we could make it. We were a Python+Flask shop, and we had candidates who claimed both those skills flesh out some API endpoints in a stripped-down sample repo. They could use their own computer, and their own editor, and collaborate with me, their potential new coworker. And I enjoyed giving the challenge, and liked making the candidate feel comfortable and excited to work on some of our code. We gave instructions ahead of time like "you're going to clone a Git repo, edit the Python, run the tests until they pass, then commit the results". You know, the absolute bare minimum you'd need to be successful at the job.
Some candidates with many years of listed experience had never used Git. That was a little surprising but maybe they'd worked at a Perforce shop or something. Nevertheless, they knew in advance they'd be using it today, and that's not exactly some obscure job skill we were asking them to learn and then forgot. OK there: "I didn't use Git at my last job but I read up on it, and I might have some questions." Not OK: "What's a clone?"
Others didn't have an editor or IDE set up. Just how? I get not taking work home with you, but you've never once written a little program to balance your checkbook or make anagrams or something else random? And again, we'd told them ahead of time they'd need it today.
It's OK not to be a Python expert, but if we give you:
def add(a, b):
return 4
and ask you to implement it, and you say you have more than zero days of experience, I do expect you to figure it out.
And I'll say it: we had plenty of people fail at the fizzbuzz step, often in astonishing ways. One memorable person couldn't grok the concept of factoring it out into a function like:
for i in range(1, 101):
print(fizzbuzz(i))
so that you could arbitrarily test fizzbuzz(1_000_000_000_000). They had the novel idea of capturing stdout and comparing it to a test fixture stringwise. Which, OK, it would work, but... I asked them how they'd see if the a test value in the trillions was successful and they mentioned using CLI tools like tail to check just the end. I did verify that they weren't just playing around for conversation's sake, like coming up with an absurd idea and exploring it for the fun of thinking through the problem. They were convinced that was the right way to write real unit tests in large codebases.
So no, I'm not giving up coding challenges, not until I see a higher passing rate among people who list years of experience on their resume. If you claim you've been writing Node apps for the last 10 years, by gosh, I want to see that you can at least write a working function in JavaScript.
So we should stop the coding challenges, stop LeetCode, stop whiteboarding, stop profiling candidates, stop asking for their GitHub page. How is hiring supposed to work then? Just post the contract online and the first one to mail it back gets the job?
I like live coding challenges, something like a ~2 hour pair programming session, ideally modifying an existing project. I invest as much time as each candidate, while we are both exploring whether we want to work together.
I personally hire people I’ve worked with before and I know what they can do.
If I can’t do that, I ask to see their prior work.
Good programmers have usually written a lot of code. Having them walk through and explain it usually gives a good idea of what they can do and how they think.
Sometimes you meet a programmer who only works on proprietary code and can’t share it.
In that case I ask them to explain the design of something similar, and write a modestly sized example of a component of that system. Watching them in their own dev environment for 30 minutes usually tells you all you need to know.
You could also, you know, talk to people, like we did before Google and the Silicon Valley bro-wanna-bes decided that coding interviews was the only solution.
I've hired plenty of people without having coding challenges or any form of live coding. I've been happy with all of those hires. A former co-worker did some take-home coding tasks, which they'd then talk to the candidate about during the interview. I feel like those hires where worse in may ways, they certainly didn't stay around as long. That may very well be completely unrelated obviously.
For years people have been complaining that exams aren't realistic, that some talented people just don't do well in an exam situation and we've mostly come to the consensus that this is correct and mistakenly filter out highly talented people. So why wouldn't we apply the same logic to hiring?
If you're hiring for a specialist position some coding exercise can absolutely be in order, but I can't think of any reason to have them for a junior position. So I wouldn't recommend dropping coding tasks completely, but I'd apply them more selectively, otherwise you risk missing a number of really good hires.
Part of it may also be that so many companies and interviewers are absolutely terrible at doing coding challenges, but do them anyway, because Google and Facebook do them.
I admit I've been hired once from just a friendly chit-chat, but I can't see how this could be reliable. There are so many bullshitters in the industry, who, during character creation put all their skill points into Charisma. They are smooth talking, good looking, and have all those charming Ivy League mannerisms of an Investment Banker or Enterprise Sales person. And while they might not be able to be technically deep enough to fool an engineer interviewer, they will absolutely fool the director or VP who does the final screen and has the final say. These kinds of people flock to companies who don't do robust technical screens.
No, there are other ways, especially as AI coding assistants become more capable and developers can be more productive if they are able to leverage them.
One approach that I've encountered (with a YC company) is that the first interview was actually a code review. One of the founders asked me to review some SQL DDL, some backend API endpoints. The DDL was missing some indices that were needed for the queries. It was using an integer ID field. The API endpoints were missing input validation, error handling, etc.
I thought this was a GREAT way to start an interview that tested for depth of experience and platform/language knowledge.
This actually inspired me to build https://coderev.app because the tooling for this felt like it would be clumsy for both the interviewer and it was certainly for me as the interviewee.
But a lot of times, seeing a candidate's portfolio -- if they have one -- is probably even more insightful than any coding exercise. When I've been on the hiring side, one of my favorite things to do is to look through a candidate's GH and ask them questions about projects they've done, why they chose specific technologies, etc.
All the time. 300-400k SLOC in C++. Legacy in the sense that there were no tests of any kind. Little-to-no documentation. Solo developer at the tiny company. Fix bugs and add features while keeping the system available to the tens of thousands of users.
A more recent example: here’s a patch for a critical feature we need. It was written over a year ago. The original author isn’t available anymore. You can write the code from scratch or try to resurrect the patch against master.
Being able to jump into a project and lead people towards some goal is definitely a skill for senior developer positions. Yes, you generally have a team you can lean on and have the ability to do research and all that. But how do you show that you can do all that in an interview?
Agree with the conclusion that a good thing to test for is for problem-solving.
The tech side depends a lot on what you’re doing. Although it gets ridiculous and organizations get lazy with this part. You don’t need to be white boarding graph algorithms for a junior web developer role. If your application is a social networking role and you’re interviewing a senior developer or architect? Definitely. They’re going to be teaching this stuff and need to understand it at a deep level.
Heck, I have to debug the stuff that some idiot (me) wrote six months ago and it might as well have been someone else who wrote it for all I remember about how to debug it
Edit: apparently this wasn't written clearly enough that I was the origin dev that I'm having words with???
Last week I was debugging some similarly uncommented, undocumented, Go code from 2020 written by lunatics that is also a critical piece of the system.
It hurts.
Where are these dev jobs where _don't_ have to figure out some mysterious issue in a barely maintained GitHub repo semi-regularly?
Hall, I do greenfield embedded programming. Most of the code I touch is completely new; the entire codebase has been replaced and rewritten by me over the last three years. Not one other developer has contributed any significant amount of code. Even then, in this scenario, I'm still reverse engineering some arcane undocumented code at least once a week. Nobody ever documents their libraries properly so I have to rip everything apart to see how it works. Or I need to adapt some vendor example code or a GitHub repo into a slightly different form. Even just existing in the ESP-IDF environment I have to dig into the core code to figure out what the hell it's doing and where these error messages come from.
If you can't read someone else's Gordian knot of undocumented code, I'd argue you are probably not very good at writing code. Reading bad code is a vital part of learning to write good code, and is generally just a fundamental core skill that every programmer needs to have.
Imo unreadable and undecipherabe code is one subset of bad code.
IMHO the willingness to "dig your way down" to solve a problem in the correct layer (rather than working around it by adding post-hoc ameliorations in "your" code) is one of the major differentiators between junior and senior engineers.
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
Not just all the time — right now in the other browser tabs I'm ignoring in favor of this one lol
> Fix bugs and add features while keeping the system available to the tens of thousands of users.
Don't tens of thousands of users warrant more developers? Or having enough of a budget to work on tests, or other things to improve the developer experience and be able to work without lots of stress? That's unfortunate.
So not necessarily any budget for more than they did.
So, definitely, I always need help debugging any kind of code (unless it's rather code that doesn't deal with product features).
> All the time.
So you were forced to do it in a silo? No access to any type of documentation, whatsoever? Not able to refer to any C++ manuals or any other type of documentation? Not permitted to access the internet to look things up? Barred from using online forums to seek help?
The OP says: "A “4-hour” assignment". This is take-home, so candidates are free to use any C++ manuals, documentation, AI, online forums, whatever...
A partner of a friend quit their job earlier this year. They then took 4-6 weeks to prepare for each interview with Big Tech companies (4-6 weeks for Meta, 4-6 weeks for Stripe, etc.). Along the way, they also took random interviews just to practice and build muscle memory. They would grind leetcode several hours a day after researching which questions were likely to be encountered at each Big Tech.
This paid off and they accepted an offer for L6/staff at a MAANG.
Talked to them this week (haven't even started the new role) and they've already forgotten the details of most of what was practiced. They said that the hardest part was studying for the system design portion because they did not have experience with system design...but now made staff eng. at a MAANG. IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Wild; absolutely wild and I feel like explains a lot of the boom and bust hiring cycles. When I watch some of the system design interview prep videos, it's just a script. You'll go into the call and all you need to do is largely follow the script. It doesn't matter if you've actually designed similar or more complex systems; the point of the system design interview is apparently "do you know the script"?
Watch these two back to back at 2x speed and marvel at how much of this is executed like a script:
- https://www.youtube.com/watch?v=4_qu1F9BXow
- https://www.youtube.com/watch?v=_K-eupuDVEc
One of my hires was a physicist who didn't know any of the jargon and had a look of terror when she first started on our whiteboard problems. Once we led her to a comfortable spot and she got into it she started talking about tools she made to help with her physics work with zero dev knowledge (she was shocked when we called her tools software, she was like, but it wasn't REAL software, we were like hate to break it to you, but that was not only real software you wrote, but crazy impressive). The white board problems were a tool to highlight potential.
I get that these big companies just want drop in widgets not people and that is what they are searching for, I just think it's funny they are using something that was created for the exact opposite purpose.
Part of me is kind of glad I'm too old to be one of the cool kids and have no hope to land a job that does these sorts of code challenges.
Of course 95%+ of it will usually be useless in your real-life work, to solve most of those problems in the time given you need to know the problem and the optimal solution to it so basically memorization with little thinking.
In other words, anybody, regardless of what university they went to or what courses they took out what advantages or disadvantages they had, can learn this stuff in a couple of months if they have what it takes for the role. And because the skills are so standardized, the process is pretty differently objective.
It's not expected that they'll actually use the specific skills in the job. But it shows they can learn skills of that type and then perform them at a high level in a stressful interview situation. Which is a great signal for whether they can learn the skills needed for the specific project they wind up on and perform in a high stakes deadline scenario.
I'm not defending the system, but I am saying there's a clear logic behind it.
You know you've designed more complex systems. The interview has no way of knowing if that's true, or you're an actor following the "I've designed complex systems" script.
And acting talent is arguably more common among the population than programming talent.
Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Also they say they've forgotten. But if they were able to get that position, they probably could do medium level leetcode problems. So, I'd doubt they've forgotten all of it. I'm pretty sure they'd still be able to solve easy level problems, which most people can't solve off the cuff. They also probably still know the complexity of a bunch of essential backend operations (search, sort, array and hash lookups, tree & graph traversals, etc).
The point is not that the friend didn't pick up some implicit knowledge or become a sharper engineer than they were before grinding, it's that by exploiting the screening strategy, they got placed into a job they're not truly qualified for.
Are they bright enough to fake it until they make it? Maybe, but that's not going to be the case for many of the countless placements that were made like this, and hints at why both product and software quality is in bad shape these days.
But on the other hand, it's also not a very strong signal for my years of experience working on and designing portions of systems. I've never actually done "design a system from scratch", because every system that is big and complex evolved from some smaller and less complex system.
So the kinds of questions that come up in these system design interviews are best answered in real life with "I dunno, I'd pick something simple and refactor it later if it becomes a bottleneck".
"What kind of database would you choose for this?" "Whatever we're already using in production for other things." "How would you model this data?" "As simply as possible, so that we can get it out fast and start learning things."
But these are not interesting answers in these interviews. I have usually found myself saying "well, in real life, I would advocate for simplicity and iteration, but for the purposes of this interview, I'm happy to discuss potential trade-offs in technology selection, architecture, and data modeling".
But I think it's easier to study up on the "right" answers to the architecture and modeling questions, without spending a lot of time learning how to start simple and when to evolve toward complexity, etc.
Doing more of the design up front is important in a big tech environment almost all the time because:
a) many different possible infrastructure (eg. databases) are already at your fingertips, so the operational cost to using a new one is a lot lower. Also, cross-organisation standardisation is usually more important than this marginal extra local operational cost.
b) scale makes a big difference. Usually good system design questions are explicit about the scale. It is a bit different to systems where the growth might be more organic.
c) iteration is probably way more expensive for production facing systems than in a smaller company, because everything is behind gradual rollouts (both manual and automated) and migration can be pretty involved. E.g. changing anything at all will take 1 week minimum and usually more. That’s not to say there is no iteration, but it is usually a result of an external stimulus (scale change, new feature, unmodelled dependency etc), rather than iteration towards an MVP.
Now, a lot of this is pretty implicit and it is hard to understand the environment unless you’ve worked in a different company of the same scale. But just wanted to point out that there is a reason that it is the way it is.
Source: SRE at Google
Perhaps.
I had 9 years of practical experience myself before going through that grind (leetcode + system design) some years ago. It is no substitute, but you don't come out of it the same engineer. You read, write and understand code differently.
Also, I have to admit that with the new perspective, there's a reason why I'd probably hire the newbie who's done this over the experienced engineer who hasn't (and maybe continues to dismiss it). The two gaps are not equivalent. In my experience the newbie will probably cover many of the necessary bases on the job fairly quickly with mentorship. I can also appreciate that 9 years of practical experience didn't help me cover DS&A or system design. I doubt that any kind of mentorship would've helped me either. Learning this stuff is a grind, plain and simple.
A newbie will also make mistakes that I can understand. I can see them pick a heap, or a linked list because it's the algorithmic sound choice for the problem. Then I can point out that for the specific use case, an array might not be as efficient, but it's a good trade-off to simplify the code. They will understand this. But an engineer that indiscriminately reaches for arrays and hash for any situation is a different story. Exponential inner loops for big searches, huge recursive calls. Those things show up in code written by experienced coders IRL. You click on a button and wait 5 seconds for the response.
I'm not sure who's faking it until they make it, but based on my own experience, I'd say that perhaps I successfully faked it for 9 years.
If the role is correctly staffed, nobody should have to be looking over that person's shoulder questioning data structure choices and algorithm efficiency regardless of what the person's background is.
It's absolutely the case that there are people with 9 years of experience who are not prepared to meet the needs of that role. There are people with 19 and 29 and 49 years of experience that fit into that bucket. Practice alone doesn't get you there.
But as you note, as a rule, you expect your bright, leetcode-grinding "newbie" to need mentoring and oversight to compensate for their propensity to rely on shallow theory where they don't know better from experience. That's not someone a team wants in a L6/staff role either!
For high level roles, you need bright, curious people who are experienced but not rigid. Admittedly, there's only so many of them, but when you start having to fill those roles with people that don't meet each of those criteria, you can't be surprised when the team's morale and output suffer. Which is exactly what we see as an outcome of the recent boom (and each prior one), because you end up having the blind lead the blind through the minefield of engineering.. and that's going to be messy no matter how clever or book-studied those folk happen to be.
The thing is, those things are easily fixable after the fact. A bad architecture is not, and without practical experience designing systems in the real world, you don't have those instincts to guide you towards the correct choices because you haven't made any mistakes to learn from.
> L6/staff
> higher-level engineers
Traditionally, one would work up to senior roles over the course of one's career, often by pursuing internal opportunities to learn and exercise new skills within one's current role before applying for new roles that rely on those now-practiced skills.
Placing into senior roles because you did well at Stanford, spent a couple years writing CRUD apps, and then grinded puzzles for a couple months is inevitably something that happens when the industry needs to fill more senior roles than there are engineers with suitable experience, but it's not a healthy phenomenon and definitely shouldn't be treated as the norm.
Having done this prep, I can't tell if this comment is sarcasm. Building real systems makes you a good engineer. Maintaining systems over a long period of time makes you a good engineer. Working with other experienced engineers makes you a good engineer.
Doing this prep you do learn a few things along the way, but it's contrived. You already know what you need to learn, which is often the hard part on the job. It's works as a filter since the ability to learn concepts is important, but it's usefulness continues to trend downwards as it becomes more standardized.
https://news.ycombinator.com/item?id=37985450
(Note I am saying people who "believe" they cannot get a good job. This would be people who worry a lot, people with unusual experiences that other companies avoid, and under-performers who got fired. I am sure there will be some great hires in those groups, but likely less than during regular hiring)
Are you suggesting that any coding interviews and challenges are simply removed from the existing processes? That just means you end up with more candidates to choose from, which doesn't sound helpful at all if your goal is to end up with better hires as the comment above was suggesting.
No, the hiring managers that are into cargoogle culting are not actually that interested in how to do interviewing properly. Not unless, say, google does it and they can copy it.
For them the important thing is that leetcode is a safe, defensible choice because "everyone else does it that way".
This is based on a real system I designed and have been maintaining for ~5 years, and is also very similar to other systems I've run at previous jobs.
About half the candidates complain that it's not a realistic question.
I think this is exactly what a lot of companies try to do when interviewing. Depending on how much time they want candidates and interviewers to spend on the task, this ranges from leetcode-style problems to bigger coding challenges (possibly with some debugging or collaboration involved).
What would you suggest concretely?
We all part from the same place (understand me: from the standpoint of the hirer, I know we have different vital stories). They open the position to all of us and want to be impressed. The one that makes it, gets the job.
It is the best way? Absolutely not. Is it unfair? No.
All crucial skills in the modern technology or legal workplace.
This is what people say whenever one criticizes the methodology of any test. They say that the real test was actually the friends we made along the way. The real test was the degree to which you are willing to humiliate yourself, or to accomplish things you don't understand the purpose of, or to accomplish things that you do understand the purpose of but disagree with logically or morally.
We filter for the worst, most damaged, and most desperate people.
These are probably the best candidates, from the perspective of a manager who is concerned with stability and not rocking the boat, at the cost of results and workplace quality.
The problem with this story is that people need to work with and rely on this person for responsibilities they're not yet qualified to meet. In some cases, a "smart person" will be able to grow into the role, but the road there is long and messy, which becomes a frustration for colleagues, supervisors, clients, users etc.
Because of the prolonged boom we just went through, the industry -- especially at FAANG's -- is now saturated with smart, naive people trying to fake it until they make it, leading to a gross decline in quality and consistency compared to where we have been and might otherwise be.
The bummer is that you're right, it actually is worth even this much investment, because these companies do pay extremely well. But it's still horrendously inefficient, because the companies are getting a very small improvement to their signal to noise ratio, at this great cost (which, notably, they don't bear).
Someone working at billion dollar VC funded outfit should understand the cause of making smart but speculative bets.
BTW, I have personally made this bet in the past (though I didn't put 3 to 4 months into it...) and it was indeed an excellent bet to make. But benefitting from a system does not preclude identifying problems with it.
if you provide utility to the market, you shouldn’t need credentials or a corporate ladder to climb to prove it, it should be immediate compensation no matter how disparate
so despite how coveted tech compensation packages are at this tier of company, a seemingly smart person getting a good job after doing a contrived aptitude test does meet that criteria. other people outside the field (and within) have difficulty passing it and don't have the cognitive ability to study for it, or the financial stability to prioritize studying for it
I also agree that a less contrived aptitude test would be better
In my mind that's a rather nasty practice.
Not that I'm complaining. I'm happy to pick up people that are good at computers but wouldn't be able to pass that hurdle, and probably wouldn't hire anyone that has.
The most surprising thing when asked about their experience was that most of the details have already been lost before even starting the role.
I’m never really surprised that there are so many security vulnerabilities out there. Tons of software is completely half-assed: slow, error-prone and inefficient. The market supports it because VCs keep pumping out companies that throw spaghetti at the wall until something sticks. And it has had an influence on the kinds of skills we optimize for at the hiring level.
The problem I see isn’t that we’re trying to hire better developers. It’s that the tests we use are misaligned. Getting into a MAANG with all these hazing rituals and then you get stuck resizing the corners on buttons or editing XML configuration for some ancient system is a waste of time.
But so is hiring programmers that only understand frameworks.
Nobody hires anybody other than artists based on whether they can "create an artful and _____ ____". People are hired based on either what the have done, can claim to be able to do, or whom they know.
But yeah in general I agree with you – I just don't think it's necessarily always so clearly a waste of time.
Demonstrating the ability to learn, even if it's just scripted, should be good enough. Granted, they might get passed over by someone with actual experience, but the lack of real experience shouldn't be a deal breaker. It's fine as long as the employer has realistic expectations of the employee.
People generally don't even want you to go deep on anything, but you're can't stay to shallow.
It's just bsing, yes your absolutely right, it feels like memorizing a script and saying the right words at the right time
If I were interviewer, I'd probably do a coding assignment that builds on an already existing mini codebase, quarter or half day assignment (2-4 hours), with a strict time limit to turn it in by the end of the allotted time. It would test for someone's ability to read unfamiliar codebases, with the codebase being reasonably small for the little time that's available, and the ability to build features on top of it, to write clean readable code compatible with the codebase, etc. And an a 30-60 minute more open-ended informal knowledge interview to gauge how knowledgeable the candidate is and where their strengths and weaknesses are. And previous work would probably be a good indicator, like open source work and/or previous job experience.
1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.
2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."
3. Submit a code sample of pre-existing work.
Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.
I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!
I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring client, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.
Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM.
As an interviewer, you really can't win.
- If you have a take-home test, people will either complain that it's too involved, and time consuming.
- If you do whiteboarding, people will claim that it's not representative of the actual job.
- If you do paired programming, people will claim it's unfair to those who don't test well under pressure.
You're hiring for a senior role, not a junior or mid.
Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.
It's like asking a doctor to go over basic human anatomy.
They need to be able to compare candidates, and senior devs will be expected to code. Their goal is to pick the best candidate. Asking them to demonstrate their coding skill isn’t just fair game, it’s what half the interviewees actually want. A lot of people prefer being evaluated on code than on soft skills (just read the threads here for tons of evidence.)
It’s useful to know whether a candidate is going to be on the principal engineer path or management path. It’s useful to know whether they are actually good at coding, and how good exactly. And it’s also useful to know if they are willing to do what needs to be done once hired. Someone interviewing for a senior coding position who refuses to code during the interview is about as big of a red flag as you can have, and I’ve been part of hiring teams that will politely excuse someone for that, and I agree with the reasoning.
This might not make sense until you’re on the hiring side of things, but having a resume does not entitle one to being hired for a higher title and more money.
I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
I know this, because I used to look at resumes and have long conversations before I ran my pairing interview. And I found the results almost uncorrelated. Now I save the fun conversations for people who make it through the screening. But by then I already know if I want to hire them.
My pairing interview (it's a pairing session, not a code challenge) covers aspects of engineering that I specifically care about. It's not terribly hard. But at the end I'll have a pretty good idea if I want to turn this person loose in my codebase.
Your candidates aren't fresh out of school or just starting their careers. They already have plenty of work experience. They already know how to code and have been doing it for years. A conversation will tell you much more about their approaches to problems solving, team work, and mentoring than a coding challenge will. Looking at their GitHub, their work experience and references will tell you the rest.
It's like asking a doctor to go over basic human anatomy.
I'm telling you from my own experience that this is observably not true. Twice now I've been burned by internal hires who objectively had years of experience (easy to verify when they have been working within my division) and yet couldn't code their way out of a paper bag.
And tangentially, with some of the health care experiences I've had I very well might try to administer a basic anatomy quiz to potential doctors if I thought I could get away with it.
People say "No I don't have any past hobby work, never contributed to OSS, and all my professional work is verboten to show" that's not unusual, but it's also not an encouraging sign to me when interviewing.
Every time I find some open source I would contribute to, what I wanted to implement is already there or is already in the works. I'm not going to change shit just to change it. I'll buy them a coffee tip and leave.
I have had to do this, on numerous occasions. Sometimes there is a system that no one who built it is left, the documentation doesn't exist or can't be found, and that's it. This is not a great example. Being able to debug a program, follow requests around and see what's happening is a good skill.
What this example is really bad at though is that you pigeonhole yourself into developers. If you have a legacy php application, you are weeding out the java, python, ruby, go etc developers that do have those skills but don't know php and will be slower at it. Despite them possibly being stronger developers given a couple months of hands on php.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
This I agree with. Suggested time can easily be gamed, so someone who has more time can look like a better candidate.
I'm pretty sure the thing they had me debug was a real bug they had hit in an open source dependency. It seemed like they had hit the bug, taken a snapshot of the code at that version, and written a test to exercise it. (And presumably then they fixed it, separately.) So it felt a lot like bugs I run into in the wild, but with some of the hard debugging work out of the way (getting it to the point of an easy repro in a unit test) in order to time-box it. I really thrived in this interview. It may be the only interview I've ever done where I felt like I was just using my actual skills like I would during the work day.
Sounds like an OK test to me. Great (senior) developers should be able to do that kind of thing. Categorizing yourself exclusively as "a Ruby developer" is a career trap.
And a lucrative one at that.
It's ridiculous how developers mindlessly accept that you should constantly be learning to keep yourself relevant, but keep it shallow by just jumping from one tool to another, instead of encouraging deeper knowledge of generalizable patterns that stay relevant across waves of technological disruption.
Be a generalist. Yes, deep specialists exist and yes, some of them have successful careers based on their deep specialty, but betting on specialization is like a high school kid planning to be a MLB baseball player.
No engineer that makes 7 figures calls themselves a ruby developer with the exception of DHH.
Assuming that high earners are offsetting that to the higher end, most people aren't making 6 figures, and the bar isn't which language they're programming in.
$132,270
This means half of all full time employed devs are higher, and half are lower. The mean is more skewed by higher earners but is similar:
$138,110
It also varies quite widely by geographic location, from a mean high of $173,780 in California to only $125,890 in Texas, from $199,800 in San Jose to $132,500 in Austin to $98,960 in rural Kansas (where I have actually developed software before!)
The short of it is, the vast majority of software developers do not make the top salaries. Even L6 is rare within the top tier of tech. There is a lot of delusion in this field around pay, so it's important to be well informed. As a field we are still very well paid compared to most other jobs especially considering our safe working conditions and lack of needed credentials and education. Compared to most of the work on this planet, it's still a goldmine.
Source: https://www.bls.gov/oes/current/oes151252.htm
Junior salaries go down much lower than $100k.
Congratulations, your experience is limited. The BLS stats represent the actual US salary data, not just your limited experience. If you want to make a claim about salaries in the US then look at data across the US and not just whatever is true within your limited bubble.
> And forgive me if forwarding the BLS statistics to candidates doesn't get them to accept offers
Did I ever even suggest such a thing?
> Did I ever even suggest such a thing?
My point is that the BLS doesn't set market rates or report on them.
Sounds like a good deal to me.
The key here is not to categorize as a “language developer”.
Consistent total compensation offers of $1mil+ is probably reserved for roles above these, sometimes called Principal or Distinguished. I would say the rate of having one of these in an org are like 1 for ~150 engineers.
How many companies are out there paying $1,000,000/year for devs?
How many devs who can command that kind salary are going to put up with bullshit coding challenges?
Google alone has 4000 directors all making a million at minimum.
I have only worked at FAANG across my nearly decade long career. So, it’s a biased but very large sample.
That's not unheard of, but it's certainly rare. $1 million+ is not a "typical salary", even at Google.
Debugging old DSL vendor specific languages or code so old using, frameworks and standards long out of fashion and support, that they are half way to being a different language.
Adding support for some back ported features or patching security holes in an old client or legacy stacks.
Or at a big company we had some escrow code from a much smaller partner that we ended up becoming responsible for.
Often getting the environment setup for proper debugging is more work than anything.
But yes, it's a good test for a senior+.
Know what I'd do if the interviewer asked me to debug PHP? Pretty much return the question:
"I've never used PHP. Are there logging macros/functions defined somewhere? Where do I see the output? syslog maybe? Is there a debugger of some sort I can use? How do I run each `piece` of code in isolation?"
(I am assuming the job listing did not explicitly mention PHP experience. If it did, both myself and the recruiter would absolutely deserve to fail me for this interview).
Interviewer: Now reverse this array.
Me: OK, in Python that would be array.reverse(), or reversed(array). I bet JS has one of those, probably the .reverse method.
Interviewer: Great guess!
That was genuinely fun. I came out of it feeling like I'd learned a few things, and the other person got to see how I'd reason about a new problem.
It was a pair programming exercise and so with some help from the interviewer and the IDE I was able to fumble through to a working result. I agree it was fun and educational.
If you've actually worked at a large company, you'd know that 90% of the real work is done by like 5% of people (maybe even less). If the interviews worked this ratio would be so much better.
And frankly, if they're able to adapt a random leetcode problem to be able to solve the coding test that I give them, then that's exactly the kind of adaptability that I'm looking for in a prospective software engineer.
- Cracking the coding interview.
- Elements of programming interviews in Java|Python|whatever...
- leetcode & other sides with paid premium subscriptions...
- mock interview bootcamps...
It's no longer about skill, it's only about gaming the system.
I’m serious. It’s not even guaranteed you’ll get the job if you do the above but everyone who passed the big tech interviews will acknowledge they fucking studied for it. What do expect here?
If you do the above prep work and pass the multiple interview loops you’re pretty smart honestly. The vast majority couldn’t do it even with all the study possible and even the smartest couldn't do it without any study at all. The bar is pretty high.
TFA: Coding interviews are a standard way of hiring.
Also you: “Too many people in the software development industry don’t know what they’re doing.”
The lack of cognitive dissonance is remarkable.
GP is of the opinion that coding interviews filter out people who can't code.
How can we even ban them using LLMs in their "ML-leetcode" problem when the problem is on LLM tokenization or something?
Like, if I lose a candidate because they're experts at using cursor + claude to do their coding (i.e. they solve it by writing 3 prompts and filling in 2 errors which are easily spotted in a total of 10 minutes), despite them having NeurIPS caliber publications and open source code, did I filter out a fake grifter, or did I filter a top tier candidate?
I just don't buy that leetcode style problems make any sense for anyone whose doing anything involving LLMs, and like it or not, increasingly large amounts of the cushy 500K/yr FAANG+ jobs will involve LLMs going forward.
Literally every day this week.
Documentation? The code was last updated in June, the documentation was last updated in January - of 2022.
The team? The half that didn't get laid off two years ago all quit for greener pastures. Sure, there's a team that owns this project - along with three others, all of which they've contributed maybe a dozen lines to over the past year, and 11 of those were dependency updates.
The issues began when we started working together in a Data Engineering team. Most of our stack was based on Hadoop, Kubernetes, Ruby, Python, and several other technologies that required a basic understanding of cloud computing.
However, since we had such a wide range of work experiences and backgrounds, I often found myself not doing the work I was hired for but instead covering for others. I ended up doing a lot of "glue work" to compensate for the fact that some colleagues couldn’t even handle basic tasks, such as Python CLI packaging using Docker or inspecting jobs in Hadoop. After some time, I decided to leave—not because I thought I was special, but because I was fed up with spending 80% of my time providing support for people who were hired at the same level or even higher than me.
I recognize that the company had very poor hiring practices, but some form of basic technical testing is necessary.
They were unfit to complete the tasks. But again was not their fault, I think was the hiring that failed.
One concrete example was the fact that our pipelines was quite straightforward for data engineers: packaged Ruby and Python CLIs that runs commands. The runtime was k8s. One of the biggest issues were when something broke in production, none of the people couldn’t go to the container, check the logs and understand the failure.
There’s one situation where I think makes sense to hiring without a test: if the company has the resources and money to provide levelling training for all new hires with no exceptions. I do not know how practical it is.
But having a simple coding task does some great things in that it will make the interviewee have to fix problems. They need to ask when unsure. You'll know who stops and asks and who plows through and solves the wrong problem. You'll know who throws their hands up when instructions are unclear and just says "I can't do this" and gives up. It's one of those extremely valuable and very effective filters that you just shouldn't be without.
The actual assignment can be however simple, but if they manage to install the tools, clone your repo, read the instructions, build the thing etc. then you already filtered out half the applicants or more. And that's without even starting to code.
The fun part of take-home assignments is that there's a moral hazard incentive to spend extra amounts of time on an assignment even if they say "only spend X amount of hours on it," even lying amount the amount of time spent if the interviewing company asks.
More than once I have completed a take-home assignment with emphasis on sticking in the timeframe they recommend but still solving the problem adequately, and then I get chewed out in the followup interview "why didn't you implement X?"
Relatedly, a massively open-ended take-home assignment is a bad take-home assignment since it favors those who have more free time to be thorough. (shoutout to the take-home data science assignments which ask for the final deliverable to be a PowerPoint presentation)
Sure, how? There are no perfect processes, only drawbacks that you can live with.
I also find the idea that interviewing is somehow exploitative of the interviewee ("feel like working unpaid shifts for a job they don’t even have yet") to be somewhat bewildering. It's not like your coding challenge is useful work for company that you are doing for free, in most cases interviewing is also a huge time sink for the company as well. Not every interaction should be quid pro quo...
1. Applicants getting hired because the manager and the candidate is from the same institute or they like the same team or band.
2. Bias based on age or race. I try to remove my own sub conscious bias as best as I can but most don't.
3. HR thinking you are not good enough and rejecting because you did java 18 and not java 19. Or you can write impressive React but haven't done Vue.
It takes the managers and HR a little out of the equation. They just rubber stamp your decision.
Take home exercises are something we tried but the incidents of cheating were so high and good code is so subjective that we gave up.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
> If the job requires specific tech skills, test those skills
Sounds like someone failed a coding challenge.
In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with. Telling them to get rid of it without attempting to understand what problem they're trying to solve and without offering alternatives, is, to say the least, not a productive use of anyone's time.
I find in software development we try to reinvent the wheel all too often instead of borrowing practices from other professions. Let's have a look at what civil engineers have to go through to get hired at half the salary of a software dev, and let's incorporate that into our practice. That will make everyone a happy trooper !
Unlikely. It's usually what the most senior person either read about or experienced in the past. In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
They just did it like everyone else because they believe exactly what you do -- that someone somewhere created the process with intention.
I feel like we've so lost the plot with tech hiring that we settled on what appears at best to be a local maxima.
I'm told this is still the common style of interview for mechanical engineers, which says something about what it's like to work in that industry too.
Agree, such or similar unrealistic interviews are loss for both sides, plus they are plain STUPID. I've performed ~400 interviews that most included coding in my career with consistency and standardized report (can compare apple to apple).
Having to spend time to do some work before an interview to prep (like a presentation about your experience or something) seems reasonable, but a company should be able to test technical skills adequately in the 5+ hours they spend interviewing candidates.
I feel like the take home assignment trend is part of the backlash against doing coding problems live in interviews and I agree with the article that it's just worse. It takes more of your time and also opens things up to candidates cheating even more than live coding over zoom does.
Just stick with the live coding and ask good questions.
Relevant part: the interviewer/CEO's philosophy is not to code challenge because no one properly vetted for "problem-solving, collaboration, and growth in relevant areas" (tfa) would ever accept a job they were going to fail at. Will always stick with me.
Now, here's the weirdest interview of my life: An acquaintance who runs a software business interviewed me by bringing me along to an NBA game he has season tickets for. The interview in the dining area before the game was normal, the game included some breaks for more interview discussion, but the home team had the best shooting night of their entire season so the crowd was very rowdy and we each had 2 beers.
"Are you testing the candidate on what the job will actually require, and ONLY that?"
With an onsite whiteboarding challenge, you're most likely testing if they're really good at thinking and talking in front of a crowd when extremely nervous, not writing code.
Same with a live coding challenge over zoom (CoderPad style) - this tends to be a test of nerves more than coding ability. But, if you're planning on a lot of pair programming, and want them to be able to dive into that on day one, maybe it's the right way to interview.
I find take-homes often the most accurate way to interview, as it most closely reflects the reality of most coding jobs. You're on your own, you don't have to deal with the extremely odd (and for some of us terrifying) feeling of someone watching you type. Hopefully, the take home questions are carefully designed to reflect what the job requirements will be. As almost every other comment has pointed out, debugging an ancient code base without documentation may be a perfect question, or it may be completely irrelevant and unnecessarily eliminate candidates.
But please, make the interview match the requirements you're actually hiring for, otherwise, everyone's time is being wasted.
As a hiring manager, I strive to evaluate for potential and adaptability, not a highly specific list of skills. I think many candidates prefer that, too, that they will be allowed and encouraged to learn how to do this job. I do not want to hire people that don’t want to learn or can’t do things they haven’t done before. And I may or may not even know exactly what the job will require. Do not assume your interviewer can tell you exactly what will happen once hired — they can’t! One thing I do know, for sure, is that the job will require communication with others, adaptability to changing requirements, and a willingness to learn the company’s workflow. I need to make sure those boxes are checked, sometimes more than I need to see how well someone knows jQuery or CUDA or sorting algorithms.
It depends on what kind of functionality we're talking about, but this kind of task is exactly what people at my current startup have been assigned at times. It is absolutely possible to build a CRUD web app with reactive UI using modern tools in a few hours.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
Again it depends on what the debugging task is. At every startup I've worked at, it's expected that an engineer is able to jump into a task that they know very little about. Granted it becomes less reasonable the more niche the task, but PHP and Ruby are not particularly far apart in skillsets in the grand scheme of things. I would expect any web engineer to be able to do this.
> Hiring processes should focus on problem-solving, collaboration, and growth in relevant areas
I agree with this. And, hiring should also focus on technical ability which does include working through difficult and unknown problems by oneself.
Yes, but it also leads to tons of bikeshedding. "Should I implement a linter? Should I write unit tests? Integration tests? Should I mock HTTP calls or use a mock server? How should I deploy things? How do I structure my CI?" All these choices are extremely subjective and context dependent and then somebody's gonna pass on you because they don't like that you used Cucumber.
Setting up a project is not something you do often, and if you do, you probably have templates and some team standards.
It's much better to give somebody a project structure and ask them to implement some basic task in it.
But how often do devs normally set up a project from scratch? We have like 3 new apps in a few years where I work now in addition to the long running ones. So on average one in like hundred devs here have been part of creating something from scratch.
Sure, when you know what you're doing it's quick. But the first time can be slow, no matter how good you are. So for a coding task that should take a few hours, you might have spent all the time just getting up and running, and then you actually start the task as over time.
We are blessed.
With software you never have to stop proving yourself, and your skillset is always a few years away from being outdated. A doctor with 20 years of experience would be welcomed anywhere, but an engineer with 20 years experience is viewed with trepidation. The next "big thing" could roll out at anytime and suddenly crypto engineers are getting 800k job offers so everyone furiously tries to learn crypto stuff. A few years later that all dries up and now you have 100k engineers who are out of work and learned tech that no one cares about anymore. All the LLM engineers might be in the same position next year.
Which doctor would you prefer do surgery on your brain - the one that jumped through all the hoops and went to 15yrs of med school - or the one that learned brain surgery through a YouTube tutorial?
I think you know the answer. The problem is not gatekeeping in general it’s the detail and nuance of how gates are being kept and how to find the right amount of gatekeeping.
My answer is that I don't have access to what I'd really like to base that judgment upon: their success rate, adjusted for the difficulty of the procedure.
Simpler, less-risky procedures would therefore have a lower skill/experience bar.
Basically, I'd want someone with decades of experiences to remove a tumor deep inside my brain, but I'd be much more willing to use someone who learned via YouTube this time last year if I'd suffered a TBI and was experiencing swelling and intra-cranial pressure.
No unnecessary gatekeeping, but maximum necessary quality standards (which I distinguish from gatekeeping).
Practically? No one has figured out how. The best we have are coding challenges.
It's not a perfect system, but it's a lot harder to fake being an experienced doctor than it is an experienced developer.
Didn't have to solve PDE's on the whiteboard, or regurgitate integral transformations.
The insane thing would be to expect someone to take you at your word for 150k*, when hiring managers know that 50-70% of people with your resume fail to write a nested for loop.
Anyway, in lieu of a coding interview, I opened up a couple of tickets (one a long term change request, another a real life problem we encountered just the day before). I provided them logs and answered questions and watched the interviewees move through both scenarios.
We capped the time at an hour. The focus was more on the interaction between them and myself—it actually used the "real stuff" they'd encounter.
I was amazed at the diversity of the responses and how differently people approached a problem.
Pretty much all the time. I also found that debugging things without asking for too much help was a great way to learn the systems involved, which eventually made me a better and more useful programmer, with more domain knowledge.
Granted not everyone has the luxury to spend time doing this, but I don't think it's that rare. In many cases it's even what companies want you to do, when that documentation or help is simply unavailable.
Every time I get those I request to do a live coding assessment, if they are not willing to lose even 1h of their time they were not serious about you.
If you decide your process really requires a solo assignment you can offer a gift card to prove you appreciate the candidate (and don't be stingy, you are already saving a lot of money by not having 1 of your engineers interview)
What gives?
If you really want me as a developer, pay me. As an industry, we might even set a standard rate for these homework assignments. I suggest a rate which is enough for me to say it wasn’t a waste of my time.
As an anecdote, I was interviewing at a startup that sent me one of these homework assignments. I told them I would do it for $1000 now, or whenever I got around to it otherwise. I never got around to it.
The hiring manager might hd really liked a candidate and conclude that a coding challenge just shows that the new hire would need extra time to ramp up.
Yet, ignoring money, you could excel in the coding part and the hiring manager might not like you because you lack "X" (e.g. "communication skills", "team culture", etc), which in reality is a way to put they just didn't like you or there were not really looking for someone to deliver, example, they were looking for someone to play politics or that might benefit the hiring manager more in the long term.
Not always the more technically capable candidate gets the job.
Can people game them? Are they flawed? Absolutely. But that's true of literally any interview practice.
The idea behind whiteboard interviews is if they're challenging enough, the only people who can fully exploit them are either very bright, very motivated, or some combination of both. That's a better signal than asking someone to lie about their "passion," which any slob can do.
This is pretty much my day job. OP is spoiled if they think this is an uncommon situation.
If you are a hiring manager and would like to experiment with something like this, please get in touch!
[1] https://pencilbeam.com
1. Said challenges should reflect the work done on a day to day basis, not something completely disconnected from the company as a whole
Google's interview challenges should be very different from say, a local web development agency's interview challenges. I've seen too many companies where either the challenge was way too difficult/complex for the job on offer (build a complex modern app from scratch for a junior/graduate level role, or solve a bunch of leetcode tests and HackerRank puzzles for a WordPress agency job), or where it was far too simple (I remember at least one software engineering interview for a React focused role where the challenge could be solved without writing a single line of JavaScript).
2. The requirements and environment should match a realistic working day
When was the last time you had no access to the internet, no access to an offline IDE, no access to premade software, no one to help, a boss standing screaming at you over your shoulder and a camera tracking your every eye movement in case you're 'cheating'?
Probably never right? Unless your company environment is a dystopian hellhole, this is probably not your typical set of working conditions. Don't expect interviewees to work under them, they're not kids in school doing an exam.
3. And that said projects should be somewhat realistic in terms of scope
Most people aren't coding a new website/app from the ground up every day/week. Expecting someone to create that in a few hours feels incredibly unrealistic, and an incredibly poor method of judging someone's skills in the role.
One asked for a 20+ page personality assessment (perhaps completing that _was_ the test?). The other asked for an interesting technical challenge which I was happy to do -- for fun -- and then followed up with a request to write a basic web service which was... unexpectedly simplistic for the level of role I was applying for.
I pointed out a number of my public, solely maintained repos on Github illustrating exactly the same sort of thing they were looking for and turned down the followup assignment, but we decided it wasn't a good mutual fit.
* Some companies design take-home assignments that mirror challenges their team has spent years refining, setting unrealistic expectations for candidates to match or exceed that work. Talent is sometimes dismissed over minor differences, like testing styles or even coverage percent.
* Working with people is about compromise. When passionate people collaborate on a creative endeavor, there's always some back-and-forth of ideas. But in these assignments, often only the unilateral opinion of the evaluator matters. I guess being rejected from a monoculture early in the process may actually be a blessing in disguise.
* Many companies fail to provide specific, actionable feedback after candidates spend hours on assignments.
I don't mind take-home assignments, but I avoid investing excessive time in them, as it’s not sustainable and often has low ROI.
Yesterday! Working on a large project means there’s always some issue with a dark corner no one has looked at recently and because there’s no team to support, I get to go and bug hunt.
Today, privilege means starting from no real job experience to writing enshitified_prod_tokenizer() and laughing at the poor sap having to fix it in a few years.
All the time now that WFH is normal and collaborating with a team member is 10x harder and 10x slower.
This is a feature not a bug. Companies are testing if you can focus and complete a hard uncomfortable challenging task, because at your job you’re expected to do things you don’t want to do, but will be rewarded for doing.
That is an option if you have higher personal obligations and are struggling mentally.
Also do they really want to set the tone that “I expect you to intuit what I want and I won’t tell you directly”? Sounds like an awful place to work, right out of the gate.
Maybe? non-desperate folks might waste the company's time. Usually companies can only give out one offer at a time. If the non-desperate person takes a while to accept, someone else in their pipeline may get an offer and flake.
We had a candidate agree to a start date and then cancel 2 weeks before because they decided to stay at their job.
> “I expect you to intuit what I want and I won’t tell you directly”?
They do tell you directly? "Do this hard uncomfortable task, where the task is study leetcode, completing a coding project."
I never deliberately snuck ambiguities into coding challenges. I've used it in oral sessions for senior and above devs though. "I'm a product manager who wants to add a new feature. I don't really know what's involved but I want you to implement it. Feel free to ask me a million questions and we'll talk it through!"
I have only once had this be a problem and in a way, it was my fault for not noting the ambiguity in the submission. Every other time I have dropped a comment saying "you could have intended A, but also could have intended B so this is when I go back to product and request more information."
Yes, complain all you want about what companies SHOULDN'T do. But then what do you feel is a better approach?
From my experience, you ask 4 software engineers what a proper interview should look like and you'll get 4 different answers.
This has literally been my job for a year. That is an actually important skill to have.
$100 is insultingly low for a 4 hour assignment.
If we assume 15 days off that leaves roughly 245 workdays per year and with a salary of $200k that would be close to $800.
A fair amount of compensation to spend half a weekend on a project would be $400.
That's exactly my problem with take-home code challenges. If I go to an interview, and they waste my time, they have to waste their own. But with a take-home assignment, they can waste my time without wasting their own. That may lead them to be more wasteful of my time.
Debugging (and setting up the debugger) is a core skill every developer should have.
Otherwise it's "well yes it's not great but it's better that not doing it"
Applicants underestimate the cost of a bad hire.
It's common for candidates who have an internal referral to skip all these steps, because the referral establishes a base line of credibility.
These assignments are so much closer to a simulation of doing the actual job than the industry standard LC interview. The only thing closer is work trial which has the significant problem of requiring candidates to not currently have a job.
Literally my first internship. Had to migrate a small system from C to (modern) C++ in the span of 6 months. Most files (and I'm going to say in the ballpark of 85%) had last been touched in 2003.
I started my intership in 2020.
In all seriousness, we just need to get together and refuse to do these types of problems (starting with hiring managers and strong candidates).
do you show them a quick'n'dirty solution that ignores edge cases but shows i'm a pragmatic and not going to overcomplicate things?
OR
do you show something fancy that you'd never actually do in an real codebase that shows off my depth of knowledge and where my ceiling is?
I would rather do this than some tired ass LeetCode problem.
It's one thing where companies are applying this to 1000 applicants for an entry level position, but they also use this tactic when they reach out to you and tell you how excited they are about your background and resume.
In reality they didn't evaluate your background or resume and are just cold-calling you to fill some portion of their "funnel", then ignore all their excitement and reasons they reached out to you and have you jump through random hoops.
I can't claim to be always be good at it, but this is what I strive for as well. I like talking to a candidate about something they are experienced or even expert at, and asking questions until I find the boundary of their ability. It can be very illuminating what happens when they hit it -- some get defensive, some get enthusiastic.
Defensiveness might just mean they're being interviewed and I failed to adequately put them at ease. But usually when we're neck-deep in some specific topic, both of us will kind of forget that it's an interview -- especially since the topic is more in their area of strength than mine.
Otherwise, defensiveness often means they never actually wanted to understand the problem they were working on, they just wanted to use it to get a degree or ship something by throwing it over the wall and forgetting about it. Even someone who is heartily sick of their PhD topic (as in, everyone who is over 1/3 of the way to getting one) will be relieved to discuss the ideas behind it, the reasons why it caught their interest in the first place, when they don't have to do the work of coming up with rigid results and writing them up.
Enthusiasm is usually a good sign, though even there I have to watch out for excessive enthusiasm where they care more about the problem itself than the benefits of solving the problem, and are likely to waste resources in unnecessary pursuits of perfection.
Oh, and finding the limits of someone's knowledge doesn't require a genius, which is fortunate since I am decidedly not one. 3-year olds can do it just by endlessly asking "why?" You'll probably need to be a little more sophisticated than that, which is good since you'll be able to evaluate their ability to explain things to you in the process.
I do like the return key, though.
How is a coding challenge not problem solving?
Collaboration can be part of coding challenges.
How in the heck do you plan to measure growth in a hiring process?
Uh.. just about every company i've worked at in the last 20 years has had little to no documentation and the guy who last touched the code left already.
I've had to do this at some level in every job.
Somebody left. The code was left to rot, whether it was a separate codebase or it was an entirely separate project. I got assigned to fix it. Sure, I had a team, but nobody knew more than I did, so it was just other devs who might have another perspective.
Collaboration and support might be standard, but mostly in the form of other smart people, not always documentation, up to date software, or people who know anything about the code.
I personally prefer these kinds of challenges as they mean I don't need to maintain two skillsets.
Lol, I'm pretty sure that's been my job description for the past 20 years.
No coding challenges.
No take-home assignments.
What is the alternative?
Is this a joke?
My answer to that question: I'm doing it right now, and have had to do it at least occasionally in almost every job I've ever had.
"Thank you for sending that along. I would like to withdraw my application for this position."
Do it anyway.
And if they don’t want to do it, we don’t want them.
Right the fuck now. That's what happen when you get saddled with whatever monstrosity the last coder(s) who quit commited or had to maintain themselves.
And then you get the joys of trying to frankenstein some modern js "component" developed like only React exists on some vanilla / jquery front. Also a personal rant for those modern js script kiddies: forms action attributes exist and work.
What are references for anyway...
The question made sense: how do you add to very large numbers together? I assumed the key knowledge they wanted was that the numbers had to be represented as strings and you should use bc_math to compute them. So I answered that, and they said, "Yeah, great. Now write a function to do that." I sat there for a few moments thinking about it and realized it was just nuts. So I asked if I could just end the interview process. They were clearly shocked, I doubt many people end an interview process on the last question. I said that this wasn't realistic and that if I did that for real it would get code-reviewed to hell. One of them kept saying but it's to see how you would do it if bc_math wasn't installed. I simply stated "I would have them install it" to which one of the interviewers instinctively nodded since that's the real-world answer.
It seemed like it would have been a good place to work up to that point. I didn't like my current job at the time, it paid considerably more, had more public holidays due to weird German reasons, etc.
I thought it was a hard but fair question. I still feel bad about not doing well. It wasn't all that close to what I'd be doing in the position, but it wasn't unrelated. Perhaps if it were a PHP role I'd be more bothered? But even there, it seems like one way to check if someone understands what's going on under the hood, so as to not naively do expensive stuff without realizing it. The fact that one should not write their own library, and that if you tried you should get code reviewed to hell, isn't relevant if that's the purpose.
For example, it's not uncommon for a challenge in finance to be "build an exchange matching engine that takes orders, modifications and cancels from different threads in two hours." That's totally doable if you just fly through whatever comes to the top of your head, and maybe have an hour left over to test, debug and modify it. But that project (an exchange matching engine) is something that the CME has 10 full time programmers working all the time on. A more realistic 2 hour task is something like "look at edge cases for the order cancel type, and figure out a way to handle malformed inputs." Then you can actually write production quality code for them to judge.
Most finish it, some don't and still get hired. It's a workalong in which we see how they approach the problem, how they discuss it with me, and what they do if they get stuck. We tell them to go ahead and google things, just tell us what they're looking for.
Over 10 years, we've got a pretty good record of hires that are smart, professional, and effective. And without specifically trying to hire for diversity, we've had a surprisingly diverse range of hires. We don't care about "culture fit", it's more like "work fit".
So I agree that the sort of code challenge where you send them away and see what they come up with is both unfair and a poor indicator. The real hiring test in an interview should be "can I work alongside this person?"
Some candidates with many years of listed experience had never used Git. That was a little surprising but maybe they'd worked at a Perforce shop or something. Nevertheless, they knew in advance they'd be using it today, and that's not exactly some obscure job skill we were asking them to learn and then forgot. OK there: "I didn't use Git at my last job but I read up on it, and I might have some questions." Not OK: "What's a clone?"
Others didn't have an editor or IDE set up. Just how? I get not taking work home with you, but you've never once written a little program to balance your checkbook or make anagrams or something else random? And again, we'd told them ahead of time they'd need it today.
It's OK not to be a Python expert, but if we give you:
and ask you to implement it, and you say you have more than zero days of experience, I do expect you to figure it out.And I'll say it: we had plenty of people fail at the fizzbuzz step, often in astonishing ways. One memorable person couldn't grok the concept of factoring it out into a function like:
so that you could arbitrarily test fizzbuzz(1_000_000_000_000). They had the novel idea of capturing stdout and comparing it to a test fixture stringwise. Which, OK, it would work, but... I asked them how they'd see if the a test value in the trillions was successful and they mentioned using CLI tools like tail to check just the end. I did verify that they weren't just playing around for conversation's sake, like coming up with an absurd idea and exploring it for the fun of thinking through the problem. They were convinced that was the right way to write real unit tests in large codebases.So no, I'm not giving up coding challenges, not until I see a higher passing rate among people who list years of experience on their resume. If you claim you've been writing Node apps for the last 10 years, by gosh, I want to see that you can at least write a working function in JavaScript.
I like live coding challenges, something like a ~2 hour pair programming session, ideally modifying an existing project. I invest as much time as each candidate, while we are both exploring whether we want to work together.
If I can’t do that, I ask to see their prior work.
Good programmers have usually written a lot of code. Having them walk through and explain it usually gives a good idea of what they can do and how they think.
Sometimes you meet a programmer who only works on proprietary code and can’t share it.
In that case I ask them to explain the design of something similar, and write a modestly sized example of a component of that system. Watching them in their own dev environment for 30 minutes usually tells you all you need to know.
I've hired plenty of people without having coding challenges or any form of live coding. I've been happy with all of those hires. A former co-worker did some take-home coding tasks, which they'd then talk to the candidate about during the interview. I feel like those hires where worse in may ways, they certainly didn't stay around as long. That may very well be completely unrelated obviously.
For years people have been complaining that exams aren't realistic, that some talented people just don't do well in an exam situation and we've mostly come to the consensus that this is correct and mistakenly filter out highly talented people. So why wouldn't we apply the same logic to hiring?
If you're hiring for a specialist position some coding exercise can absolutely be in order, but I can't think of any reason to have them for a junior position. So I wouldn't recommend dropping coding tasks completely, but I'd apply them more selectively, otherwise you risk missing a number of really good hires.
Part of it may also be that so many companies and interviewers are absolutely terrible at doing coding challenges, but do them anyway, because Google and Facebook do them.
One approach that I've encountered (with a YC company) is that the first interview was actually a code review. One of the founders asked me to review some SQL DDL, some backend API endpoints. The DDL was missing some indices that were needed for the queries. It was using an integer ID field. The API endpoints were missing input validation, error handling, etc.
I thought this was a GREAT way to start an interview that tested for depth of experience and platform/language knowledge.
This actually inspired me to build https://coderev.app because the tooling for this felt like it would be clumsy for both the interviewer and it was certainly for me as the interviewee.
But a lot of times, seeing a candidate's portfolio -- if they have one -- is probably even more insightful than any coding exercise. When I've been on the hiring side, one of my favorite things to do is to look through a candidate's GH and ask them questions about projects they've done, why they chose specific technologies, etc.
You're being tested on your general intelligence, not because anybody cares about binary trees or reversing linked lists or whatever.
A lot of interviewers missed that memo. They are in fact very focused on a specific algorithm or design approach. It's quite tiresome.