Things we talk about
- 00:11 - Patrick McKenzie Introduction
- 01:10 - Hiring Developers and the Standard Developer Interview
- 06:05 - Company Onboarding Processes
- 10:13 - Leaving a Company
- 15:54 - Referrals, Lead Generation, and Recruiting
- 22:58 - Developer Evangelism as a Sales Tactic
- 27:36 - Doing the Interviewing Process Correctly
- [Link to] Rubric
- Digital Gold: Bitcoin and the Inside Story of the Misfits and Millionaires Trying to Reinvent Money
- Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent
ALI: Hello, everyone, and welcome to another episode of the CTO Show. Today I have with me a very special guest, Patrick McKenzie.
PATRICK: Hi-de-ho, everybody. My name is Patrick McKenzie - better known as patio11 on the internets.
ALI: For the perhaps three or four listeners who have never heard of Patrick McKenzie - Patrick, could you give us a very short introduction to the patio11 story so far.
PATRICK: Sure. I've run a series of small software businesses from Japan for the last ten years or so. I've been blogging extensively through that and, for reasons that are probably outside the scope of the podcast, people actually seem to like what I blog about/comment on/tweet, so for some reason people trust what I have to say about things and I guess that's life in a nutshell.
ALI: Sure. From my point of view, I knew Patrick from afar for a lot longer than we've been acquaintances in real life. So basically, Patrick’s done a lot of writing about the career path of being a developer and perhaps starting a business and that sort of career trajectory. I think his writing was quite influential for me when I was starting out - it helped me take my first steps into business - so to then now see that I’m running a podcast where I’m speaking to him is a little bit surreal for me so this is very exciting. I follow Patrick on Twitter and he, a couple of days ago, unleashed an epic tweet storm, ranting about the state of how we do hiring for developers in our industry and so I thought I’ve got to get Patrick on the show just so we can rant about this for 45 minutes because I am screaming about it. I'm kind of holding my tongue with all my other guests so today I’m going to let all of that out. Patrick, in your estimation, what are the ways in which we mess up hiring developers in our industry?
PATRICK: I think the industry is just fundamentally un-serious about the way we go about hiring developers from the jump. So the context for the tweet storm, which will probably be linked up to the show notes, is that if we did sales like we do hiring the sales team would be fired after two weeks because they'd be clearly jokers and I gave 45 examples of that but, without recounting the entire tweet storm, because that would be a very boring podcast, I think that we largely hire, both the process of identifying candidates, bringing them in, interviewing them and making decisions on who we hire and the HR issues involved in that, like figuring out what their offer is and then dealing with them during and after the process of becoming an employee, is largely based on folklore rather than based on any sort of rational business process. I think it's conducted by people who are wildly underqualified. The people who are wildly underqualified are given anomalously high positions in the organisation and enjoy them and enjoy, in some cases, the feeling of power and authority that this gives them without any rational business reason as to why they should have that power and authority. I think that occasionally we give power and authority to people who neither want nor need nor sought the power and authority and have no responsibility in actually exercising it. A great example there is a very common practice in engineering teams which is to give any engineer the capability of blackballing a candidate, which is absolutely insane. You didn't select your three years of experience engineer with an eye towards their perceptive eye for team dynamics and you don't ask their opinion about anything else, like 'Hey, before we file taxes for this year, Junior Engineer No. 4, do you want to entirely blackball any tax strategy we're taking?' No, you leave that to the accountants because you're not insane.
ALI: Yes. That’s fantastic.
PATRICK: I could keep going.
ALI: Please do, please do. I am taking notes and I’m going to follow up on a lot of things you're saying. Continue.
PATRICK: Sure, so I think the standard developer interview is, well, many things about the standard experience of developers in the industry are more akin to fraternity hazing than they are to any rational business process. The standard interview which we give some variance of is how many ping pong balls fit in a school bus, another is 'Here's a blackboard, do something hard under conditions of stress on the blackboard, despite the fact that we're not hiring you to do anything that involves blackboards,' are more akin to a hazing ritual than they are to actually determining engineering talent or IQ or any conceivable thing that you would actually want to hire for. I think that we overly value pedigree to an insulting degree in the industry and then it's crazy that the left hand doesn’t know what the right hand is doing - the left hand is incredibly interesting in pedigree, both in terms of educational pedigree and institutional pedigree, like 'Did you just come off a job at Google? Have you presented at the right conferences?' etc. and the right hand is wondering why more women, minorities and other underrepresented groups aren't showing up late in the funnel.
ALI: I think the kind of process you're describing, perhaps our experiences are slightly different, but for me it's not just these situations where there's this kind of a HR department with a lot of power and authority that enjoy that power, it's also people who are actually quite well-meaning but don't realise the damage they're doing and I think a lot of people I’ve spoken to in the past want to do the right thing when it comes to hiring but are so caught up in their own cognitive biases and their own vision of themselves as people who are good judges of character and using all of those biases and their gut feeling about it that they make terrible decisions when they hire and then, once they've made those decisions, they then backwards rationalise that this person was a good 'cultural fit', which is another one of my bugbears, that entire concept of cultural fit kind of drives me insane, which I imagine it does you do, based on the scare quotes you've put around it in your tweets.
PATRICK: So for those of you who don't know either of us on a personal level, we've both lived in Japan a substantial amount of time and I think we've both had experience working in Japanese organisations so we've seen the pointy end of the stick of cultural fit used actually to mean fitting into a business culture and the notion that I would affirmatively make decisions at my business for forcing my employees to go through that seems deeply irrational to me.
ALI: So you've talked about, I think, the general structure of how this toxic hiring organisation exists in the type of companies we're talking about and you've talked about some of the downsides of interviewing. I also want to get your opinion on what you think of the way companies generally handle everything after that, so let's say somebody managed to get through this process where their pedigree is assessed and they go through this hazing ritual and they've been attracted to this job by options and ping pong tables and beers at your desk or whatever. Let's say they get through all of that - then we have the on-boarding process. What do you think about that, as it's executed in your experience?
PATRICK: On-boarding processes are all over the map, from companies which have an actual document at the company or perhaps even people whose job it is to ensure the on-boarding goes smoothly. What we're talking about with on-boarding here is a combination of legal/paperwork issues that have to happen and tactical things like, this is going to sound stupid, but making sure that a person who needs a laptop to do their job actually has a laptop arrive at their desk, ideally the first day at their job, and if not on the first day, at least not, say, six weeks after they join the company. So there's that sort of issue and then there's the complicated issue of 'Okay, we have an engineer, we've assessed them to be good an engineering, they have a laptop, we've issued them logins to the various systems but there's still some X factor that needs to happen before they're capable of doing actual productive work on behalf of the organisation,' whether that's pairing with another engineer to learn the layout of that company's systems, whether it's just gaining a level of trust that they won't be breaking things, or even 'What project am I assigned to? What am I expected to do in the first few weeks? What am I allowed to do in the first few weeks? What is the process at this company, which is different from every other company I’ve worked for in my life?' Or going from 'Okay, I’ve got Eclipse open, I'm coding, I have a command line that is available to it - what happens next between empty command line and this getting deployed to production?' and all of that comes into the general heading of on-boarding and companies are all over the map with regards to how organised they are at that. Some start-ups say 'Well, start-up life, yo! You've got to just figure it out for yourself.'
ALI: Wear many hats, etc.
PATRICK: Right, and what happens when you ask an engineer with two years of experience to figure it out for yourself is, fairly predictably, they take down production and you have nobody to blame but yourself when that happens. Other companies have, let's say, engineered processes where someone gets a two week boot camp and Name of Company U, 'We'll teach you, this is the correct way to do a deploy, this is the company's one blessed way to do a code review, this is some of the Argo that develops, here is the chat bot that you interact with to do a deploy, yadda, yadda, yadda,' and while two weeks of a heavily structured process might or might not feel like a wonderful experience for someone through every minute of going through it, that seems to be strictly dominating the no-process, cowboy everything.
ALI: One I’ve heard of actually get flown out to another country and stay in Name of Company U for, I think it's six weeks, rather than two weeks, the one I’ve heard of at least, but yeah, I’m not going to mention any names. I think that's probably the ideal but you need to be at a certain scale of company before you can implement something that big, I think.
PATRICK: But you certainly don't need to be at a terribly sophisticated scale or have millions of dollars BC to do something like 'Okay, we're going to take out an iPhone and record one of our engineering leaders doing this with somebody for six hours, which is how much we invested in the interview in the first place, and then we're going to transcribe that recording and put it in a Google Doc and find out which parts are important and which parts are not and then have the next recording reflect more of the important stuff and less of the non-important stuff.' That's an investment of a few hundred dollars and a few hours of time and if you’re not investing a few hundred dollars and a few hours of time into making someone productive in their first six weeks at the company then you're not running a business.
ALI: Yeah, I mean the basic arithmetic of that just is insane so, yes, I understand. We've talked about interviewing, we've talked about on boarding - the other thing that I think, especially when you see the recruiting process as a form of sales, obviously people talk a lot about what it's like to actually work at a company, office politics and processes and all the rest of it, but what I specifically want to bring this conversation to is what happens when you leave a company - when you eventually do leave, because people leave jobs for various reasons - how have you seen that or have heard of that go south from the point of view of companies and how they mess up the hiring process?
PATRICK: I think we're dealing with a cultural script that sees leaving a company as a failure on the part of the person leaving the company or possibly on the part of both parties, rather than the natural up-growth of the economy that we live in. I would like to see companies come to a realisation that it's 2016 and the typical employee tenure is in the order of three years, or 18 months in you're in San Francisco but that's neither here nor there, but to understand that 'job hopping', it's not an anomaly and it's not a deviant behaviour but it is in fact something that is built into our cultural expectations, to leverage the fact that it's built into our cultural expectations. One example is that HubSpot, for example, has an explicit named alumni program, which is something that every software company should just straight up steal tomorrow. You're sending your agents out into the industry with the instructions to talk you up, so you should give your agents the instruction 'Talk us up, make friends, bring them back, have them come to our office and eat our food and drink our drinks,' because that has a net positive expectation for you. You should not send your agents out into the world with the instructions 'Could you please tell everyone you meet for the next 30 years of your career that we're backstabbing bastards who had you escorted out by security on your last day after you had given four years of blood, sweat and tears to the company?'
ALI: There's a book I’ve read - not recently, probably three or four months ago - called 'The Alliance' and it's written by, I forget the exact name, but one of the people in the C Suite at LinkedIn, and one of the sections of the book is about this alumni program idea they have but they also kind of really focus on this idea that three to four years is kind of the natural length of your tenure at a particular job and instead of running away from that they kind of say 'Hey, look, we know it's going to be about three or four years total so let's sit down now and try to figure out what we can do for you, what tasks we can assign you, what responsibilities we can give you to help you grow the most possible and help you do whatever it is you want to do after those three days,' they sit down and they kind of acknowledge this. Now, when I’ve seen the book mentioned in other press, people kind of construe it in the most negative way possible, in that it's a way for start-ups or companies to impose short term contracts on their employees, so I don't think it's been seen in a very good light in the reviews I’ve seen of it, but it has helped me focus on the short to mid-term goals of my employees and say 'Look, I want to help you achieve whatever it is you want to do next,' and I think just saying those words, saying 'It's okay for you to leave and do other stuff and come back,' signals that you're not a maniac, which is a big deal in this industry when it comes to leaving a job.
PATRICK: I think that's absolutely true. I've heard stories of others unfriending people on Facebook after they leave the company because it's perceived as an act of disloyalty. We're not Japanese salarymen, folks - we all understand that this is a commercial transaction here and, like many commercial transactions, it has a start date and an end date and on the end date you give firm handshakes and you say 'We're going to be in the industry together, we'll be friends,' well, maybe friends is a strong word, but 'we'll be socially pleasant to each other in the future,' exchange handshakes and then move on to the next transaction. I also think that one does not exactly have to be capitalist tooth and claw to acknowledge the fact that employment is a business transaction, not a sacred trust. There may have bene times and places in which it was a sacred trust but we're not living in one at the moment and therefore we should be unabashed about acknowledging this and institutionally weary of anyone who is attempting to extract concessions on the basis of employment being a sacred trust.
ALI: Right, so my business is I run a Ruby on Rails agency and we really need work all the time, so from my point of view everyone I meet, everyone that I speak to, any conversation that I have, whether or not I genuinely enjoy it or find value in it, is the potential for some lead later on. So when I say goodbye to an employee I’m wondering 'Is this person going to be the CTO of a start-up that ends up being a client one day? Or are they going to refer us to someone like that?' So from my point of view, just from a pure kind of Gordon Gecko capitalistic point of view, it really does pay to be nice to people when they leave. I have experienced the exact opposite when I’ve bene an employee - I’ve been told that 'Whatever you do next is going to be terrible. We gave you this great opportunity and now you throw it back in our face,' and all the rest of it, so I think I’ve learned, from having these things thrown at me, to not be that insane.
PATRICK: Perhaps it's because of the overly youth-focused nature of our industry that folks have not seen how a ten-year engineering career plays out yet and they don't realise that you will meet the folks from year two and year eight after they've had six years of professional development. Man, I’m saying ten years as if ten years is a long time, given that we're all going to be in this industry for 40+ years unless we cut our careers short for whatever reason, which, no judgment there, there's plenty of good reasons to not be in this industry for 40 years, but I think most of us will be.
ALI: Yeah, I mean that's already happened to me. The reason I am so focused on referrals and lead gen is because I have already seen the profession where I have given a talk at a code school and I have said 'XYZ, do these things to get your first job, negotiate your salary, do these things to signal that you're good,' etc., etc., and people have emailed me and said 'That was a great talk,' and then four years later I get an email from them saying 'Oh, by the way, I’m working at this company, we need some help with security,' or something like that, and then if you look at the lifetime value of that client, the average lifetime value is around £60,000, so that means that for any potential person I’m speaking to in a talk right now, I might have to do ten talks to get it, but it's worth a total of £60,000 in revenue to the company, so it just makes sense to cultivate goodwill with people in the industry who are just starting out, knowing full well that in five or ten years' time, like you said, they're going to be at an experienced level of skill and political clout wherever they’re working.
PATRICK: Right. I have definitely seen some folks who have sent me emails back in the day when they were looking for their first intern job, etc., who, fast forward a few years, 'Oh yeah, I'm CTO at a company with 30 employees now,' or I think one of my favourite anecdotes about this is Paul Graham once had a young kid from Ireland over at his house because they bonded over the internet about Lisp and the young kid's name is Collison, a not insignificant portion of the value that Y Combinator has generated for its partners is currently based on the fact that the young kid who does Lisp did a few other things and then founded a company that does payments - hot tip for all concerned, disclaimer of a commercial relationship which will probably be published before the publication of this but you can put a link to that blog post if you want. There’s plenty of less dramatic examples of the same sort of thing. It costs nothing to be polite to people, to treat everyone that you interact with like a human and to genuinely listen to people and hear what they actually care about, to go the extra 30 seconds when someone says 'Okay, I’m looking for this thing that you're going to,' and rather than saying 'I don't do that,' say 'Let me introduce you to Keith, my buddy, who does that thing that you are looking for.'
ALI: I think it's the only way to survive a long term career. You have to be building that network and I think any touch point you have with potential employees or somebody working with you or someone who's leaving is an opportunity for you to store a little bit of goodwill with them, and basically my business right now is monetising the bank of goodwill I’ve built up over the course of my six or seven-year career as a full time developer. I couldn't survive, full stop, without, I guess, being polite to people. So, yeah, that's what it is.
PATRICK: I think that it is substantially the same for myself. I don't know if monetising is exactly the way I would phrase it because I would probably attempt to be helpful to people regardless of whether there was a return at the end of the tunnel for that but it is certainly the fact that I feel like I’m harvesting in 2016 two seeds that were planting in 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015. So, given that you do not intend to die in the next couple of years, plant some seeds today.
ALI: So yeah, just to go off-topic a little bit, my lead generation strategy now is basically to accelerate the planting of seeds so that's all I’m doing, full stop, knowing full well that they won't turn into leads right now but in a year's time, in theory, we should be stacked with work. So this is basically my entire business strategy.
PATRICK: That's another thing - and turning back to recruiting - we overly focus on the day to day stuff that is just purely transactional and don't spend enough time making systems that plant seeds for us, despite the fact that we have the superpower that allows us to do this that so few other people in the world have. So many people have interesting conversations about technical topics and yet that conversation doesn't make it into a visible artefact like a blog post or a conference talk, etc., or if it does make it into a visible artefact there's not even two minutes of thought about 'Okay, how do I go from having this artefact to having the contact details of someone interacting with it?' It takes a little work on one day of your life to set up a mailing list and say 'What are my hobbies?' Hating on Bitcoin is a hobby for me - I will say this now, I’ve been saying it for several years, but hating on Bitcoin is going to be a very lucrative hobby before the end of time - so, given that hating on Bitcoin is a hobby, I should have an email list for fellow Bitcoin haters. When I do my usual hating on Bitcoin shtick it should be, at the bottom, 'Sign up for my hate mail list for more like this.' Less sardonically, 'Sign up for my email list for more like this about software marketing and sales,' which is something that I’ve actually done for the last couple of years, which has worked out fantastically well for me. The amount of seeds planted by my blog, given that I had a mailing list attached to it since, when did I start doing that? 2013 or so, I think, greatly increased the volume at which planting happened as opposed to 2006 through 2012 when I had no mailing list available.
ALI: From the point of view of recruiting, though, I think this is something that companies can and do do - like the outreach part of recruiting where you're kind of looking for canvas. Sometimes I speak to newer entrepreneurs, people who have just started coming into this kind of work in programming, and their idea of how they're going to build a following and get people to listen to them, with the aim of hiring developers, is not very sophisticated at all. It has none of the things that we're talking about here. I've seen it been done much better, in particular by - and I have no affiliation with this company, it's just that I know some of the people that work there and I’ve seen them do very good things - a company called Future Learn in the UK, which does an online learning platform, and they spend a lot of time, they basically have at least one person full time whose entire job is to build assets like a mailing list. It's very difficult to convince a developer to write a blog post, just out of the ether, so what they do there, for example, is they pick a topic and everyone from the team has to write a paragraph about it and then they publish that and they help people work towards putting out more content. They put out a lot of talks about their development process, about the things they learn, about how to be better at your soft skills as well, and for them hiring is really easy - because they've done all of this work they have an enormous pool of highly qualified candidates to work through, to the point where I’d recommend people to this place and they turned me down because their hiring platform is absolutely packed and they have a new intake next year. So that technique has worked really, really well and been actioned by a company that's doing hiring so I was wondering if you had any other things that companies could do to improve the beginning funnel of who they attract when they're looking for jobs.
PATRICK: In broad strokes that's exactly what I would recommend. It's funny how since a few companies have shown the successful model, developer evangelism as a sales tactic is becoming more widespread in the company but no one will pick developer evangelists just to evangelise the wonderfulness of their engineering team. The math clearly works out, right? You can either pay an outsourced recruiting $120,000 for delivering you four warm leads in a year and you know that the way they got those warm leads is by spamming hundreds of people on GitHub and you don't care, you're cool with that as long as they deliver the bodies, or you could pay someone and just say 'You do Python, we do Python. Great - make it your mission in life to fly our flag at every Python watering hole you can find online and offline to let people know we are a wonderful place to do that and to help promote the engineers in our company as being awesome at their jobs to people who would not otherwise hear that.' I think that, to the extent that companies are starting to dip their toes into the water, that's one of the opportunities that they miss. Companies are very organised around tooting their own horn but they don't realise that tooting the horn of their employees is probably more instrumentally effective in getting people to attempt to have a job at that company and one of the reasons that it's effective is because, say you happen to have a very talented engineer named Sarah at your company, people will join your company not because they see Sarah having fun in the job at your company but because they see Sarah and want to work with Sarah and will follow Sarah where Sarah goes. People are ultimately loyal to people, not to institutions, that's why militaries since World War II have figured out 'Okay, we're going to put teams together, treat each other like families and not break them up for rotation kind of reasons and then they will owe their loyalty to the military because everyone they know in their family is in the military but they owe their ultimate loyalty to their band of brothers.' Similarly, I think that Sarah being a talented engineer is probably nose to the grindstone, banging out features and responding to JIRA tickets and yadda yadda during her daily life. You should definitely encourage Sarah to 'get out more' and toot her own horn a little bit, but since engineers are not normally wonderful at doing that and often won't make time to do it, or your engineering organisation won't score them on doing it, you should have your developer evangelist, or hiring developer evangelist I guess you would call it for lack of a better word, do the grunt work for them. Say, 'Sarah, that presentation you did internally was awesome. Can you forward me the PowerPoint? I will edit out the details and then post it on your blog.' That's, what, an hour of work Sarah wouldn't have made the time to do it - you should make the time to do it. You should make the time to say 'Oh, I realise that you contribute to this open source slide that we use extensively inside the company, how about we mention that fact somewhere publicly visible rather than siloed in the commit logs of a GitHub project that no one outside the company actually reads?', 'Oh, I saw you had a very interesting solution to a scaling issue that we had inside the company. Can you take 30 minutes after lunch one day and talk to me into a microphone about it? I'll get that recorded, put up the transcript on our website,' yadda yadda and put the call to action at the bottom - 'Did this sound interesting to you? If this sounds interesting, this is what we do every day. Give me a call - I would love to introduce you to the people who can get a ball rolling and talking about how you can do interesting things every day too.'
ALI: And none of the things you said, from the point of view of the person organising it, are actually hard things to do. They're purely organisational, you're taking a little bit of time out of this person's day, you're getting everything down in a format that you can show to the rest of the world and they can consume, but these are things that are already happening in your organisation - like Sarah would have had the scaling issues whether or not you decided to write a podcast about it, so you're just sitting on all of that value and a lot of this work sounds like it is just using it and monetising it so that you can use it to recruit developers.
PATRICK: Yeah, none of this is rocket science. It's not rocket science for marketing, it's not rocket science for sales and it's not rocket science for recruiting. Your company produces an enormous amount of value and most of it - I like to use the iceberg analogy - most of it is below the waterline of visibility to the public. You have just the tiniest bit of your marketing that is seen by people outside the company but the vast majority of day to day dev ops, backend hiring stuff, hiring processes, etc. stays siloed inside your organisation unless you affirmatively make efforts to make it visible to other people, so you should have someone whose brief is basically 'Get as much of our value above the waterline as possible.'
ALI: Okay, so we've talked a lot there about how do we get better ways to get people into the funnel and planting seeds and putting the value inside the company that we out in a format that people can see. Now, I want to get your opinion on how we can the actual interviewing process correctly. We said earlier that we tend to rate things like pedigree, they end up being something to do with how many ping pong balls can fit in a jar or data structures and algorithms on a blackboard. What, in your opinion, is the way that we do that right?
PATRICK: There’s this book called 'How Would You Move Mount Fuji?' which is about interview puzzles used by Microsoft. You should buy a copy of this book and then burn it. That was flippant - skills assessment is a solved problem; we have decades of academic research based on this. The answer to the solved problem is a work sample test - you should develop a reduced case of some work that you actually would have done in your organisation, reduce it so that it can be done in a respectful to the candidate amount of time, which means probably between two and eight hours, and then you write a sample solution or several sample solutions, you look at what you like about those solutions and don't like about those solutions and then - this is very important - you reduce that intuitive knowledge into a written rubric. When people come into interview you give them the work sample thing - it might include just them working on it in their code cave on their own laptop, it might include pairing them with one of your programmers, whatever you think is most representative of what you're actually hiring people to do - you take their solution, you evaluate it based on the written rubric.
ALI: Sorry - by a rubric you mean what exactly?
PATRICK: It's a word from elementary school pedagogy. Let's say you can earn a maximum of one hundred points on this assessment for writing a Rails application - five points if you remembered to do authorisation on all pages, three pages if you da-de-da, and then a list with actual numerical values tied to qualitative or quantitative statements that you can make about the artefact delivered, as opposed to things like 'Were they sufficiently passionate when they were banging on the keyboard?'
ALI: Okay, right. So you're saying a written rubric and you're assessing the solution against it?
PATRICK: Right, so a maximum of ten points awarded for, let's say, code organisation. Here's an example of code organisation which earns ten points; here is an example of code organisation which earns seven points; here is an example of code organisation which earns four points. People think this is hard - hard it might be; impossible it is not. There are written rubrics involved for things which are much more difficult to assess than ability to program Ruby on Rails programs, like for example the oral proficiency interview for Japanese, to pick one that is near and dear to our hearts - in 40 minutes you can grade someone at an incredible level of detail on their Japanese ability from literally could not successfully order a beer given a menu with the word 'beer' on it and a finger to point, all the way up to could have a sophisticated discourse about theology in Japanese without ever having studied theology in Japanese.
ALI: Wow, okay.
PATRICK: We should link to that rubric, by the way, because it's fantastic.
ALI: I think I want to do it, to be honest, if you find somebody to assess me.
PATRICK: This is a crazy thing - you can pay someone $100 to do that and they will spit out an answer within one hour of your time. It's an answer backed up by evidence and tested and comparable against yourself, myself, people who don't have our life experiences, people who have never been to Japan, people who have studied at great universities, terrible universities, etc. and it F-ing works - and our industry thinks this is impossible.
ALI: But at the same time I’m absolutely unsurprised that Japan has a test for this that is so effective.
PATRICK: It's actually the American state department which developed this because it's an Obama-fied occupation requirement that state department officers have a certain level of ability to speak the languages of their host countries.
ALI: Okay, I don't think I’m going to be able to find an official of the American state department for Japan in the UK so maybe on the next trip that I have.
PATRICK: There is a civilian organisation that offers these assessments for literally anybody who wants to pay $100 or whatever the testing costs.
ALI: Okay, I’m in. If I can find this, I’m in.
PATRICK: And there should similarly be - in my 20-year view of the industry - there should similarly be someone who, not a certification per se, but you should be able to pay $100 to some company somewhere and get a report on whether someone is capable in Ruby or not. Sadly, the internet does not have a great option for that yet but it will happen.
ALI: Okay, so we talked about that work sample test and assessing it. Let's say, for example, we were in a situation where we had four different candidates all interviewing for the same job. Aare you saying specifically - I want to corner you on this and make sure this is exactly what I think you mean - that what you would then decide to do is put them all through the same work sample test, with the very specific rubrics for all of their solutions, generate a score for them and then hire the person with the best score?
PATRICK: I'm close to a work sample test absolute, so basically that's exactly what I’m saying. To the extent that you want to override the 'Frank got an 80, Sally got a 65 but, man, I feel this X factor about Sally,' that almost all of the X factors that you can feel are probably something which is invidious to include in the process over the long-haul.
PATRICK: If you later go back to it and say 'Okay, we hired Frank with 80, Sally with a 65, but a year into working at the company Sally is clearly a much more productive employee than Frank is...'
ALI: Then your test is broken.
PATRICK: Right. Then you go back to the test and you say 'Alright, a) how is Sally a more effective employee than Frank is? and b) what evidence in Sally's solution was overlooked that would have given us that impression?' and then that goes in the rubric for future use, but I would suggest that people not attempt to rewrite the rubric for individual candidates because they're politically favoured or because 'He's my brother from college, I know he's good for it!', 'This person has great trajectory in front of them!', etc. Trajectory - a totally reasonable thing to hire for, but have that in the F-ing rubric.
ALI: The thing you mentioned there, the things that really you shouldn’t be including in your criteria sneaking their way in because you want to override the process, that is not the entire problem with inclusivity in the industry but it's a pretty big contributor, right? Because what a lot of people don't realise is that X factor, that gut feeling, is a bunch of different biological things happening at the same time, so one element of it is built up cognitive biases. To start with, we kind of prefer people who look like ourselves, like if we met someone from the exact same background that you have, the exact same kind of upbringing, then we already like that person better than the other person who got a higher or lower score, so that X factor is going to be there for that person because they happen to be like you and there's going to be less of an X factor for people who are different to you. So if people are asking 'What can we do to make our company more diverse?' it is to remove that X factor from being the decision making point and a work sample test like this, while it has the effect of hiring for exactly what you want, it also has the effect of not allowing things like that to enter into your hiring process, so that's why I think it's a really big deal from a diversity point of view as well.
PATRICK: I absolutely agree with that. To be clear, just feeling affinity for someone because they have similar circumstances is not in itself a terrible thing - you and I have a similar experience where we've both lived in Japan, we bonded about that the first time we met, we talk about it frequently - but if you were interviewing me for a position at your company or I was interviewing you for a position at my company and we kept coming back to the Japan thing in a role that that was not relevant for, that would be a) unfair to similarly situated candidates who, just for whatever reason, their personal story was not that they had lived in Japan for ten years, and b) it would be strong evidence that we were not actually selecting for the thing we thought we were selecting for. I think people should have enough capability for introspection such that they realise 'Wait, I keep attempting to introduce things that are not on my rubric into this decision making process. Let me step back for a moment here. Ah, I’m being affected by the fact that we went to the same alma mater, I’m being affected by the fact that we both play StarCraft, I’m being affected by the fact that we both have the same first name,' which, by the way, is a real thing, there's research on that.
ALI: Oh, I’m sure there is.
PATRICK: 'I'm being affected by the fact that I got a vibe. Maybe I don't feel comfortable saying what that vibe is but I’m vibing. I know I shouldn't be vibing, let's go by the rubric.'
ALI: And people might argue that this is just how human beings work and it would be completely unnatural for us to take all of our biases out of the equation and I’m pretty sure that's true, I’m sure they come through, but I’ll give you an example of, again, this is Future Learn, only because I know them quite well - basically, I gave my usual talks about career development and I followed one of the attendees on Twitter who said 'Thanks for the talk,' later and she basically did a lot of things that I said to do and about four months later I looked at her online profile and I said to myself 'Oh my God, I’ve got to take this person off the market, they clearly know what they're doing, they can write really well, I’ve got to give this person an offer, full stop.' She turned us down because she had other options but I then sent an email over because I knew Future Learn were hiring and she needed a place to work. I sent an email over to the CTO at Future Learn and said 'Look, your hiring process is fundamentally broken if this lady fails it because she is just amazing, she's vastly productive and she's got a lot of technical skill for somebody in her first ever job as a developer,' and the response I got back was 'Thanks very much, Ali, but because you've sent me this email but because you sent me this email I have now been removed from the hiring decision for her in this batch of candidates so I’m afraid I can't help with that,' and the point there is they actually actively discount things like recommendations - they anonymise applications so you just have the initials of the person and they remove the university name from it as well - and they are a thriving development team in London. A lot of companies have hire contractors on a higher daily rate - Future Learn is one of the weird companies that is kind of taking contractors who only seem to be after it for the money and convincing them to work at their company full time and they’ve got a very large team so they are doing this and they are quite successful, and they're not a huge corporation by any means, so I’ve got a pretty good example there of this working just fine so, yeah, I’m very pro using a work sample test and trying to illuminate cognitive bias from that selection process.
PATRICK: I could go either way on whether forbidding insider trading and information on a particular candidate is something that is positive for the company or not. I think that demonstrates a huge amount of courage in their convictions, which I’m generally in favour of, but then again the Valley folks that are hiring based on cultural fit have a lot of courage in their conviction on cultural fit too.
ALI: Conviction on its own without the content is, yeah...
PATRICK: Going back to the unnaturalness of this, though, I think that it is entirely the case that attempting to make decisions in a non-biased, data-driven fashion is very unnatural for humans, to which I say so what?
PATRICK: Dev ops does not come natural to us either. Creating systems that are self-healing does not come natural to us. The sales process, oh my God, does not come natural to me, it feels so incredibly crazy to me to send someone 18 emails saying 'Hey, Bob, just following up from last week. Have you had any discussions with your team based on this?', 'Hey, Bob, just following up. I wonder if you saw my earlier email from Tuesday,' blah, blah, blah. We construct processes to do this because it F-ing works and we know it F-ing works and we know when we do processes that don't do this we have comparatively worse results! Therefore, we should create hiring processes that do the thing that F-ing works!
ALI: Excellent. Okay, on that note I think I’m running short on time so, yeah, do the thing that F-ing works is the moral of this episode, I think. So can we have your picks for our listeners today?
PATRICK: So a good book that I read recently, in the hating on Bitcoin genre, is 'Digital Gold'. I'll give you a link to that because I’m not sure it was the exact right title but it's less of a hating on Bitcoin book, it's just an impressively well researched book into the history of Bitcoin. In terms of hiring books, I don't have any that I really like. The closest one to something I like is Joel Spolsky's 'Smart and Gets Things Done' which is just a look at how Fog Creek used to do things. Joel Spolsky is kind of a mentor of mine; I would have said that this book was the absolute bible ten years ago, I think I’ve grown a little to have my own likes and dislikes over the last ten years so I would not treat it as the bible anymore but perhaps an influential treatise by a skilled theologian.
ALI: Alright, okay.
PATRICK: Yeah, those would be my two for the moment.
ALI: Okay, excellent. So, yeah, we'll have those in the show notes, linked for listeners to check out. So thank you very much, Patrick. It's been a very good rant. I feel a lot better about my hiring rant quotient now so thanks for speaking with me.
PATRICK: Thanks very much for having me - and if any of you folks would like to talk to me, I love talking about this subject or any other, please feel free to email me at firstname.lastname@example.org. It's likely the fact that my new employer is going to be public knowledge before this podcast is produced - please don't email patrick@ that domain .com, that's a totally different guy. You can also reach me on Twitter @patio11. I love talking to folks in our industry. If there is something I can do to help you with spit balling some ideas on how you can do hiring better if you're a firm or if you're trying to get into the industry and you would like introductions or just somebody to give you a bit of a lay of the land I love helping people out with that - plus, planting seeds, right?
ALI: Exactly, and please do take Patrick or myself up on that exact offer. Patrick is extremely generous with his time, to a fault I would say, so he will almost definitely get back to you. He did to me, put it that way. Alright then, thank you very much for listening and we will see you again next time.