Because security matters
The Idea
External Pen Testers are useful, but only have a limited amount of time to get to know our systems and are not intimately familiar with the Ravelin codebase. On the other hand, engineers and data scientists within Ravelin know the codebase inside out (they wrote it after all), they understand the API and the expected functionality. Why not train all technical staff on Pen Testing? Not only will they gain experience with security awareness, leading to a more secure code but also have a bit of fun during War Games.
For those not familiar with War Games, the general idea is to have an attacking team and a defending team. The purpose is to test the infrastructure & application in order to discovery unknown vulnerabilities and put in place fixes. It is also useful for training staff who are required to be on-call as during war games the infrastructure can become very unstable and diagnosing issues quickly can make the difference between winning or losing . (https://en.wikipedia.org/wiki/Wargame_(hacking))
We would run our War Games Day over a single day (Monday) and have prizes awarded four days later (Friday). Giving enough time for any long running scripts to execute and for fixes to be put in place.
Training
This is obviously the most important step, the better the training the more likely we are to find security weaknesses. Ravelin is very fortunate to have an ex-pen tester from RSA on the data science team, who could deliver all the training. If this wasn’t the case, we would have used our external pen testing company to come in and deliver the training.
To make things a little easier we also asked staff to join teams that covered part of the system they were most familiar with, reducing the training overhead. For example we didn’t need to train API developers on the internal office subnet structure.
Attack Surface
We decided early on that we wanted to be as inclusive as possible, so pretty much everything was in scope. This was split into three distinct attack surfaces: All Dashboards (client facing and internal), All API Endpoints, All Networks (internal, external, office etc).
Aspects of the business that were out of scope:
- Social Engineering. https://en.wikipedia.org/wiki/Social_engineering_(security) For the first War Game, we really wanted to test the technical side of the business and ideally limit its impact to just technical staff rather than disrupting the whole company. Plus we had recently run an internal social engineering attempt on the whole company and implemented policies to detect impersonation attempts along with 2FA/U2F everywhere. This will probably be included next time. Social engineering of our suppliers was allowed, although no one actually attempted this.
- Unattended Laptops. Again same reasons as above.
- Physical Office Security. We currently have a 24/7 manned front desk, security guards, magnetic auto-locking doors, cameras, lockers etc, and although probably not perfect, with limited resources I wanted to focus the attention of the team on the technology.
- DDoS, DoS. I didn’t want to start an arms race; developers have access to our cloud computing console and no doubt could spin up enough servers to cause our systems to autoscale, in turn increasing their capacity and then having Ravelin autoscale again, all of which being billed to Ravelin. Also we sit behind cloudflare’s network as well as GCP’s Load Balancers, both offer DDoS protection and testing that would be a little pointless. However, discovery of slow api endpoint was included as this could easily lead to a legitimate client/staff member DOS-ing our infrastructure accidentally.
Teams
The formation of teams was based around interest from each engineer/data scientist for one of the three attack surfaces, I then attempted to balance the teams with both senior and more junior members. Importantly, we encouraged staff to pick the part of Ravelin’s stack they were most familiar with, hopefully leading to “better” results.
Each team could run their attack as they saw fit, some teams delegated parts of the system (within their remit) to an individual team member whilst other teams worked together in a control room fashion. Both methods worked well.
For the next War Games, we will encourage more of the control room style teams as engineers will be working on unfamiliar parts of the system.
Assessment
Four days after the War Games Day, we held the debrief. Each team presented their finding and real prizes were awarded (~£150 worth of prizes per team member of the winning team). In total 8 low vulnerabilities and 2 medium vulnerabilities were discovered, with each team finding at least a couple. Things like default password on our printer and being able to SSH into our telephones both from within the office network, to a redirect parameter being set as a GET variable in the dashboard. For judging we decided on an anonymous peer review poll based on the following criteria:
- Number of Vulnerabilities
- Severity
- PRs for Fixes
- Ease of Exploit
Our criteria being a watered down version of: https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator
Conclusion & Learnings
We found the experience exceptionally useful and would highly recommend it to every company. Ravelin will be running the next War Games in 6 months and every 6 months from then on.
Having the business bought into the process and allowing time for bugs/vulnerabilities to be fixed needs to be factored in next time. Although most of the bugs found were fixed during the war games week, we still have a couple of none critical bugs that still need to be fixed.
Having prizes of real value isn’t 100% necessary; however, is a good idea as it goes a little way to represent the real business value.
Next time round, some engineers will work on parts of the application they are least familiar with; this should lead to a smaller Bus Factor (https://en.wikipedia.org/wiki/Bus_factor) but also allow for working within teams that don’t normally work together.