Some of us drilled it, some tried it, some lived it. All of us lost — because in this case, the future was predestined.
The Challenge of Strategic Hiring
I’ve never gone through many job interviews myself. I have, however, conducted hundreds. Almost all were for Software Engineering roles and many for product development where ill-defined problems were the norm and innovation was not optional.
But even though I was always looking for someone who could face the challenge with a decent amount of creativity, they weren’t responsible for decision-making and setting a path beyond the technical realm. Engineers were expected to challenge it, propose alternatives, stress the system, and execute but never accountable for its future.
Hiring Software Engineers isn’t easy and anyone reading this already knows that. Still, we have tools. We can reasonably evaluate problem-solving skills, software design, technical depth, and even learning capacity. If both interviewer and interviewee are honest, many of these competencies can be observed, tested, and discussed.
For strategic and decision-making roles, however, we get into a different dimension entirely.
Don’t get me wrong. We know some of what to look for (or we fool ourselves into believing that we do). Depending on context, seniority, and organisational maturity, we expect familiarity with budgeting, delivery flows, reporting, stakeholder management. We expect experience, even if by proxy, because no one has ever been something before becoming it.
But there’s a large grey volume of work that sits beyond the visible horizon. Work that requires navigating uncertainty, committing under incomplete information, and overcoming obstacles where no established framework quite fits. In those moments, lateral thinking doesn’t “solve” the problem, it merely allows movement against a different and sometimes larger and harder wall.
And then there is failure.
“We don’t hire people, we hire results.” True if you accept good ones and bad ones, and the latter are unavoidable when exploration, innovation, or strategy are involved.
One could even argue that many of these failures are actually excellent results since we found what to avoid but let’s skip this one for now.
The Star Trek Parallel
“To boldly go where no man has gone before” was the closing narration of the original Star Trek series. It followed the adventures of a spaceship crew exploring what lay beyond the boundaries of human knowledge.
Right off the bat every crew member was expected to be comfortable with the unknown. At any moment of their mission they would find themselves in unresolved conundrums where only the human moral and ethics matrix was guiding their decisions and often it wasn’t enough. Yet not everyone bore the same burden. Not all were subject to the last-minute decision-making process that happened right before the climax of an episode. Many were engineers and specialists, operating at the edge but within predictable roles and schedules.
On the other hand the commander class played a different game. At the helm of a life boat, 24/7, their actions would steer lives away or nearer to their demise. How the hell can you measure such decision making capabilities? How do you evaluate competence when you have to face unknown unknowns day in and day out?
The recently deceased Economics Nobel laureate Daniel Kahneman and his also deceased pal Amos Tversky explored this problem with scientific rigor. One of their findings was that you should value deliberate practice and after-action reviews: Expertise in complex environments isn’t just about experience; it’s about quality experience with feedback. Effective leader development requires a form of “critical event training” where thought processes are reviewed and compared to expert models during after-action reviews. Trainee and tenured decision makers should study history and use fictional and past scenarios as low-cost methods for deliberate practice, receiving expert and peer feedback to improve their decision capacity.
Unfortunately we can’t spend weeks and months evaluating and training every candidate for a position. Only internal candidates can benefit from such a luxury, where mid-management roles often serve as training grounds. Still, even in constrained interview processes, we can extract meaningful data points, if we choose our tools carefully.
This is where the Kobayashi Maru distress beacon shows up on our scanners ( Trekkie lingo my friends ).
The Original Kobayashi Maru
In Star Trek, Starfleet cadets are subjected to a simulation with a deceptively simple setup. A civilian vessel, the Kobayashi Maru, is stranded in neutral territory. Entering to rescue it risks triggering a larger conflict. Not entering means leaving civilians to die. If the cadet intervenes, overwhelming enemy forces appear. Defeat is guaranteed.
The general idea, within Star Trek’s social and philosophical universe, was that the challenge was unwinnable. You need to decide if you lose a significant amount of civilians that you should be protecting at all costs, or if you lose a fight and your entire crew and ship with the probability of saving the stranded civilians. Before you start seeing the exit routes, mind that during the exercise execution issues would pop up left and right, blocking you from many, if not all escape routes to the point of a future captain desperation. Lateral thinking did not break the wall, it merely revealed that many walls exist.
Key Insight: If you want to be a Commander, you will have to have the ability to lead when no roadmap exists and, ultimately, accepting failure. How you deal with it will be a deciding factor to make the cut.
What Science Says About Selection Methods
So should we include such a scenario when hiring for Product Management or other strategic roles? I would like to convince you that you should but at first glance science says no.
Decades of research on selection methods tell us that interviews should predict future job performance.
Research Foundation:
-
Palmer & Campion (1997), Personnel Psychology - A Review of Structure in the Selection Interview, state that structure in the employment interview refers to the degree of standardization in interview questions and the scoring of responses. Increasing structure generally results in higher reliability, validity, and legal defensibility.
-
Schmidt & Hunter (1998) on the Psychological Bulletin - The Validity and Utility of Selection Methods in Personnel Psychology make the case that selection procedures should be chosen based on their validity for predicting job performance. Methods that lack clear job relevance generally show lower validity.
-
From Levashina et al. (2014), again in Personnel Psychology - Structured interviews improve construct validity, we learn that structured interviews improve construct validity by reducing contamination from irrelevant factors such as interviewer biases, impression management, and idiosyncratic interpretations of responses.
Broad meta-analyses in the same time range (McDaniel et al. (1994), Schmidt & Hunter (1998), Levashina, Hartwell, Morgeson & Campion (2014)) consistently show that structured interviews, work-sample tests, cognitive ability tests, and importantly for us, well-designed situational exercises have high predictive validity…
Kohn & Dipboye (1998) - Journal of Applied Psychology - The Effects of Interview Structure on Recruiting Outcomes
Stress interviews may elicit anxiety-related reactions that are not representative of typical job performance and may unfairly disadvantage certain applicants.
…but unstructured interviews, stress interviews, and trick questions do not.
A raw Kobayashi Maru, used as a theatrical no-win trap, would fail spectacularly as an evaluation tool. It teaches only that failure is inevitable. As a data point, it is nearly useless.
But the story doesn’t end there.
The Case for Structured No-Win Scenarios
Research also shows that situational judgement and simulation-based exercises are particularly effective when assessing complex competencies such as judgement, decision-making, and adaptability, provided they are structured, job-relevant, and consistently evaluated following the conclusions that we attained previously.
Supporting Research:
-
Huffcutt & Arthur (1984) - Journal of Applied Psychology - Hunter and Hunter (1984) validity and utility of alternative predictors of job performance: Situational interview questions are effective to the extent that they elicit job-related behaviors rather than hypothetical or artificial responses.
-
Lievens, Peeters & Schollaert (2008) - Journal of Occupational and Organizational Psychology - Situational judgment tests: A review of recent research: Situational judgment and simulation-based exercises are particularly useful for assessing complex competencies such as judgment, decision-making, and adaptability as they are expressed in ambiguous work contexts.
-
Weekley & Ployhart (2006) - Personnel Psychology - Situational Judgment Tests: Simulation exercises allow candidates to demonstrate how they diagnose problems, prioritize information, and respond to changing constraints.
So I gathered some support for my hypothesis but the moment a scenario is intentionally unsolvable, the evaluation must shift from solution quality to reasoning quality. What matters is how candidates frame the problem, state and work their assumptions, reason about trade-offs, handle uncertainty, and are honest when they are losing. With this we can work with in a structured and measurable manner as long as our scenario proposals are within the job context and expectations.
So there is a path for designing no-win situational judgement tests as long as we design the scenarios and interview interactions respecting some structural tenets.
Critical Insight: The moment a scenario is intentionally unsolvable, the evaluation must shift from solution quality to reasoning quality. What matters is how candidates frame the problem, state and work their assumptions, reason about trade-offs, handle uncertainty, and are honest when they are losing.
The moment we want that the scenario becomes “unsolvable by design,” evaluation must shift from solution quality to reasoning quality.
So to avoid construct contamination with less objective factors and responses we need to explicitly define what is being evaluated (e.g., framing, trade-off reasoning, uncertainty handling). A no-win scenario must map directly to real R&D PM work (trade-offs, ambiguity), not abstract cleverness or pressure tolerance ( I committed this mistake in the past: blame Hollywood! ). If the candidate is expected to “crack” or emotionally collapse, the exercise loses validity. Finally, the scenario needs to be close to actual organizational dilemmas, not logical paradoxes or hidden traps, all present in the original Kobayashi Maru but with the caveat that future captains would face war, negotiation and advanced aliens.
And don’t forget the interviewer biases and bad habits. If the interviewer subconsciously rewards confidence, cleverness, or performative leadership, you lose validity so keeping your evaluation sheet close and fighting human instincts is also part of the exercise.
How to design your Kobayashi Marus? We’ve already covered much terrain. We just need a map for it.
Let’s call it the “Pondered Kobayashi-Maru Decision” or just PKD.
PKD Requirements: Kobayashi Maru exercise types are only defensible if:
- The exercise must be standardised in both questions and scoring
- The evaluation criteria must be explicit and limited to job-relevant constructs
- The scenario must resemble real organisational dilemmas, not logical puzzles or hidden traps
Ambiguity in content is acceptable; ambiguity in evaluation is not.
The PKD Framework
Pondered Kobayashi-Maru Decision (PKD)
Not a stunt. Not a trap. Not a stress test dressed up as philosophy.
Designing one starts by accepting a constraint: you are not testing outcomes. You are testing candidate’s judgement in an adversarial environment that will take them to a loss.
It’s not about watching candidates fail. It’s about observing how they think when success is no longer an option.
First, start thinking on the most impactful features that you need on your role. Let’s consider a research and development lead, probably running a few experiments, polishing proofs of concept to a minimal viable product and making bets in a given vertical. This is definitely living on the edge of knowledge and pushing a few boundaries while refining new findings into the next product horizon.
For a Product R&D Lead, the work is rarely about optimisation. It’s about placing bets under constraint.
Typical tensions look like this:
- Learning speed vs organisational risk
- Short-term delivery vs long-term platform health
- Stakeholder alignment vs technical truth
- Evidence quality vs decision urgency
PKDs should sit exactly at the intersection of two or more of these forces, where satisfying one explicitly damages the other.
If the problem has a clean “best practice” answer, it’s not a PKD.
PKD Scenarios in Practice
I’ll suggest a few examples but don’t accept them as recipe. Work your way first, mapping your tensions and breaking points to make them clear in the proposal. There shouldn’t be any correct answer. Just revealing ones.
📋 PKD Scenario 1: The Experiment That Worked Too Well
Setup: You lead a Product R&D team exploring a new capability adjacent to your core platform. A proof of concept, built quickly, with shortcuts, is showing unexpectedly strong traction with a pilot customer (RAGs, I’m looking at you).
At the same time, the architecture is not production-ready as expected, security has flagged multiple concerns and let’s make this even more stressful with another team depending on the same components for a regulatory deadline.
The Constraint: Leadership asks: “We have a quarter to turn this into a customer-facing MVP or else we’ll lose our time-to-market and all work is forfeit. Make it so.”
Mid-scenario blocks:
- Extra headcount is not available
- Saying “no” risks losing executive sponsorship for any future initiative
- Saying “yes” risks technical debt, security exposure, and reputational damage
What can we evaluate:
- How assumptions are surfaced
- How trade-offs are articulated, not avoided
- Whether the candidate distinguishes learning success from delivery readiness
- How they frame losses when all paths have one
📋 PKD Scenario 2: The Roadmap Collision
Setup: Your team owns an exploratory product line. Another product group owns a revenue-critical roadmap.
You found a shared dependency but your R&D work could unblock future growth while their roadmap funds the organisation now and they don’t want to compromise anything. Priority is singular. There can be only one ( sword sound here for my fellow millennials ). Both cannot be prioritised without compromise.
During the exercise:
- Stakeholder pressure is asymmetric
- Data is partial and politically charged
- Delaying either path has visible consequences
Mid-scenario twist: You are informed that whichever option you choose, the other team will publicly disagree with the decision.
What this tests:
- Decision framing under social and political pressure
- Whether the candidate seeks false consensus
- How ownership and accountability are expressed
- Comfort with visible disagreement and loss of goodwill
Remember: There is no win — only managed loss.
Implementation Guidelines
Cool stories bro but who’s going to conduct them and how?
A PKD only works if discipline exceeds creativity.
Non-negotiables:
Standardise the prompts: Every candidate gets the same information, the same constraints, the same twists.
Standardise the scoring: You are evaluating:
- Problem framing
- Assumption management
- Trade-off reasoning
- Uncertainty acknowledgement
- Ethical clarity
Make reflection part of the exercise: Ask what they would review after the decision. Leaders don’t just decide, they learn.
What to avoid:
Never reward emotional collapse and charisma. Pressure is inherent. Distress is not the signal. Don’t look for James T. Kirk alikes.
Always remember: Ambiguity in the scenario is acceptable but ambiguity in what you value is not.
The original Kobayashi Maru was about accepting failure as part of command.
A PKD is about something deeper and way harder in every aspect of our lives: how people think when success is no longer available as a strategy.
In Product R&D, that moment comes more often than we like to admit. Better to observe it deliberately than discover it by accident.