Lean Six-Sigma for Hackers
Part 1
Picture the scene: you are a 25-year old Very Cool Hacker Person (TM) or 1st class BSC/MSC/PhD in Compsci holder, and you’ve just been accepted into a cybersecurity company with an exciting name, a markov chain will probably give you “CrowdCrew” or “CyberTrace” or some such. You’re happy - you’ve actually been selected to do one of the few modern jobs that is both technically interesting and, as far as I’m concerned, ethical! (Yes, some companies might try selling TLS 1.1 as a critical vulnerability, with email over port 26 as the solution. If that’s the worst your industry does, you’re in a good industry). You’re punk rockstars, changing the world, you don’t need process, that’s for suits and Java programmers! You’ll work evenings and weekends and tell yourself you enjoy it, and sometimes you might, because again, this is actually technically interesting work. You pick up some side tasks for the business, and they’re cool too.
Then, maybe you start noticing that “exceptional” weeks become every week, and every week is now not just split between a couple of clients and reading up on this cool new CVE, it’s split between a dozen clients with ambiguous requirements, and the side task that is now critical but not actually recorded in any valuable way, and now you might in any given week have time to point a Nessus scan at a client, but it’s three days and 10 pm before you’ll have time to look at it, and you’re really considering if you want to put your name to this next report because it’s terrible, and everyone’s actually miserable.
You might, depending on social situation, have noticed your friends in finance and law or EDI workshops or whatever are similar (side note, I name drop EDI shops because they actually do function remarkably similarly to cybersecurity in a way, naming they’re both desperately trying to get the attention of a CEO, who doesn’t disagree with them, they just don’t want to spend any brain cycles on this whatsoever.)
Now at this point some of you might go off on the inherent evils of capitalism, and frankly you may well be right, but ok hippy, free love or whatever failed so you may as well make your peace with it. Maybe, if you’re a business, business improvement stuff might help. I like Lean Six Sigma, and no I can’t believe I’m advocating management consultancy stuff, but I am, so here we go.
Lean manufacturing came out of Toyota post- WW2, and was one of those “constraints breed creativity” situations - being bombed to nothing and having no resources, except ingenuity, led the Japanese automaker to really focused on what the customer needed, and get money for it right away, whereas the Americans could stock enormous warehouses with enormous numbers of parts, because the capital markets let them (a thinly-veiled criticism of modern Interest Rate Policies? Say it ain’t so!). Today it tends to be combined with Six Sigma, which is more about reducing variability. Let’s go over the concepts.
Value to the Customer
Let’s start with something important to any business that isn’t propped up by endless VC money; the customer. What do they value? If your name is Toyota, it’s a car, in the case of penetration testing, written findings. Brakes on every wheel might be negotiable (and they were), and so might be, say, severity ratings. The delivery of the product to the customer, however, is not. So, most importantly, Work In Progress needs to be treated as what it is - something that, right now, is costing you and is worth zero.
Pull Workflow
Now, let’s see how that interacts with workflow. Your platonic ideal workflow might look like this:
Sales -> Scoping & Access -> Work -> Write Up -> QA -> Delivery
Or maybe for more complex projects:
Sales -> Scoping & Access -> Work Distribution -> Work -> Write Up -> QA -> Delivery -> Aftercare
Wow, that’s a lot more stages already, isn’t it?
But your personal flow probably looks like this:
Work -> Clarify scope -> meeting -> work -> QA - -> Surprise aftercare request!
Work -> QA
Scoping & Access -> Maybe work now?
Work -> Fix broken infrastructure -> meeting to explain why you’re two weeks late
Work -> -> Dream of becoming a woodworker on the Isle of Skye.
Here’s the thing - the arrows are all going one way, so there’s no feedback. So work is piling up, that leads to context switching and bad quality, that leads to demoralisation, that leads to burnout and failure.
So instead, work backwards. Workers should PULL work from the prior stage, not PUSH.
- Delivery pulls from QA when free (practically, this stage should be fast)
- QA pulls from reporting (should also be fast)
- Reporting pulls from Work
- Work pulls from scoping
- Scoping pulls from Sales
Why does this work? Well, it works because a limits the inventory. Everything that is Work In Progress has no value, remember? And if you’re only doing one thing, it’s quicker, and if’s quicker,
Part 2
Batch size
Probably not an issue with most pentest shops, but let’s imagine QA is all done at once at the end of the month. Let’s also assume you have a new hire this month, WHICH YOU DO. A new hire is also not an exceptional situation in most shops. Now, the newbie’s probably going to screw up, right? Everyone does with the best training in the world. But your newbie’s done five reports wrong, that means five to fix. That’s worse than if they screwed up one and time to fix is doubled. So, keep batch sizes small so feedback can be tight. Tight feedback solves problems.
Minimise SKUs (aka Gordon Ramsay is Yelling At You)
Hackers are l33t, and all that, and take pride in learning any new system extremely quickly. But I’m sorry guys; you’re still better at doing something you’ve done before than something you haven’t. So every product you offer is another thing you’re potentially doing for the first time, or at least the first time in six months, at which point you’ve forgotten how to X and Y has expired, or been repurposed, and all this stuff has costs. It’s another source of Work-in-Progress, and WiP is stuff that’s costing you time (and so the business money) but that the customer does not value.
Incremental QA
If you want to be a little shocked, ask someone at Boeing how much QA a finished aircraft goes through. Actually don’t, Boeing is probably not a great example. But Airbus, Rolls Royce, General Electric, Lockheed Martin - all produce components that cannot fail and all do things similarly, with only cursory checks at the end. Why? Because they’re checking continuously as they go.
Remember what we said about Work In Progress? A car that fails QA has had every single step that costs paid for by the company, and how much is it worth? That’s right. Zero.
Conclusion
Penetration testing is, and this should be extremely obvious, a business, just one that’s extremely good at marketing otherwise thanks in part to the relatively immature talent pipeline (do you really carmaking is easier? It isn’t. But it is mature). An awareness of these things is useful. Even if you don’t do Lean - I’m sure other frameworks are available - speaking the same language as your manager helps. Yes, a manager responding to “I feel shit and overworked” with “have you tried working harder?” is terrible. Managers need to escalate as well, and they need authority to do that, but no one in a business is an island, and “my cycle time on task X is excessive” is soo much better to deal with.
Anyways, I’m done. Good luck all.