In Purple Team Part 1: Improve your defensive success with Purple Team exercises, we described the concept of Purple Team and how we execute exercises with the mnemonic Purple Team platform. We started offering Purple Team exercises to customers in 2022, and it was an immediate hit. Because of this, we have improved and scaled up the service quickly.

We're pleased with the organic growth of this service from just a general idea to something significant and easily approachable. We've learned a lot along the way and invested considerable time into professionalising Purple Team exercises to deliver them at scale to our customers.

TL;DR

If you are interested in detection and response or are responsible for this in an organisation, this blog post will explain why Purple Team exercises should be on your radar. This post discusses the valuable lessons we learned during Purple Team exercises over the past year. These are:

• Integrate the Blue Team
• Find the right tool for the job
• Use CTI and open-source information
• Automate all the things

These lessons mean we no longer hold up valuable IT security people for days. It has allowed us to scale the service and lower the entry barrier for the organisations we work with in a way we did not anticipate six months ago. Not only do we love and embrace automation wherever we can apply it, but in this case, it was necessary to successfully continue delivering Purple Team exercises at scale to our customers.

Improving upon these lessons has allowed us to pave the path for repeatable and measurable detection testing at scale. Here's what's next for Purple Team exercises by mnemonic:

• A delivery model for Purple Team exercises based on detection and response maturity
• Involving most discipline areas at mnemonic in some way, shape or form
• Integrating Purple Team exercises into our Managed Detection and Response (MDR) service

Lessons learned

If you read Part 1, you should know a little about our definition of Purple Team and the general method for executing them. What have we learned so far from running actual Purple Team exercises?

Integrate the Blue Team

The Red Team typically leads Purple Team exercises, and although the Blue Team is involved, they are often an afterthought. However, we have learned through trial and error that integrating the Blue Team into every exercise stage is crucial for the exercise's success and the organisation's improvement. Not involving the Blue Team can cause unnecessary stress and confusion, turning the exercise into a Red Team activity. To avoid this, we strongly recommend integrating the Blue Team into every stage of the test.

While unannounced exercises can be valuable in certain situations, we have found that announcing our Purple Team exercises and involving the Blue Team in every stage is more effective. This approach eases the entry barrier to performing exercises, is easier to plan and execute, and leads to less headache caused by secrecy. It also allows us to focus more on improving the organisation rather than spending time and resources on administration.

It is important to note that the goal of Purple Team exercises is not to simulate a full-scale attack in real time but to assess detection and response capabilities for individual attack techniques. Integrating the Blue Team into these exercises provides a more accurate scope of tests, allowing for more effective targeting of detection capabilities. It also provides a better ability to measure the effectiveness of specific capabilities, tools, processes, or technologies used in detection and response.

Integrating the Blue Team into every exercise stage provides many benefits, including team collaboration and learning, more accurate testing, and a better ability to measure defensive effectiveness.

Find the right tool for the job

Some of our first Purple Team exercises started, like many things in the IT world, with MS Word. We began with unarguably barebones planning, execution and reporting of the first exercises we ran. The screenshot shows the first few rows of a table in Word from a planning and execution document. It's primitive and not fantastic, but it got the job done.

This system barely lasted one or two exercises before the information required to support the Blue Team grew out of proportion. We suddenly had to report on Event IDs, process IDs, how to detect specific techniques, the status for detection across multiple teams and a range of other indicators. That led to the number of columns in our Word table growing beyond the size of a landscape mode A4 page, so we realised we had to expand. And when you have no dedicated software for a problem you are trying to solve, Excel becomes your software. This screenshot briefly shows a similar system but expanded for increased detail in the reporting. We used a single spreadsheet in Excel with sorting and filtering to plan, execute and report an entire exercise. This spreadsheet also got the job done, but with lots of headaches, not just from managing massive spreadsheets but shipping the document between different teams, keeping it in sync, etc. Our Excel template nearly went to column AA (that's when you get past columns X, Y and Z).

You may wonder why we did not immediately find a better tool for the job. The YAGNI principle ("You Aren't Gonna Need It") is a practice in software development which states that features should only be added when required. So, our approach was to start with a minimum viable offering, with limited tool support and a significant amount of manual labour. We could have comprehensively evaluated different Purple Team platforms, written an elaborate Purple Team framework and developed a full delivery model with "everything automated". But if Purple Team did not pan out, that would have been a waste for us and our customers. We saw that the organic growth of Purple Team exercises was better than trying to be perfect from day one, and this still applies. We've always been honest with our customers that this is a new service and that we're still ironing out the kinks as we go along.

So, what was the lesson learned here? MS Office is all right, but its ease of use makes it easy for eager engineers to create software through raw persistence. Necessity is the mother of invention, and there was a necessity here. If you read Part 1 of this blog post, you will see how mnemonic uses the officially licensed product Vectr to support or Purple Team platform. This has given us a much more resilient way of planning, executing and reporting campaigns and test cases. We've yet to look back at Excel and Word even once since we started using them.

Use CTI and open-source information

Initially, we tried very hard to design elaborate attack scenarios for our Purple Team exercises, mimicking real attacks. For some organisations, this was interesting and seemingly had value. It was "a good story" and had an alarming effect on the readers of the executive summary reports of the exercises we delivered. However, considerable time and resources went into designing scenarios and stories. Still, eventually, the executed test cases were largely independent, and our Purple Team exercises did not simulate any single attack or follow a kill chain either way. At best, it was a haphazard collection of attack techniques loosely fitted to cover the MITRE ATT&CK tactics from execution to Impact.

Another pitfall we identified was that Red Team operators from a penetration testing background tend to execute Red Team operations in the same way as penetration tests, with the same tools and a limited understanding of how real attackers operate. Involving our Threat Intelligence Team helped our Red Team better understand actual threats and shift from a penetration test mindset to a proper threat intelligence-based approach. By applying our Threat Intelligence Team, we have gained a deeper understanding of threat categories such as Nation State Groups (NSG), Organised Crimes Groups (OCG), and even threat actors targeting specific sectors with specific Tactics, Techniques and Procedures (TTPs) that these groups use. This approach also aligns with our Red Team Framework™ and TIBER methodology, making all the adversary simulation services we deliver, including Purple Team exercises, better and more Threat Intelligence-based.

Through working with our Threat Intelligence department, our Red Team operators have been exposed to the world of Threat Intelligence and can rigorously apply what they have learned. We've gained a much better understanding of how real threat actors operate and are getting unstuck from the penetration test bubble we have been in. We've discovered that an enormous range of machine-readable threat intelligence is readily available, and we're finding better ways to apply it. Realising that a lot of this was possible to import directly into our Purple Team platform for visualising and simulation was a remarkable finding for everyone involved. Here are some of the sources we have learned to love:

• MITRE CTI – GitHub repo containing machine-readable intelligence on threat groups, tools, etc. We've imported this into our Purple Team platform and can quickly start simulating most threat groups categorised by Mitre.
• The DFIR Report – Detailed descriptions of real intrusions by real attackers. Something we draw inspiration from when designing and planning Purple Team exercises.
• Atomic Red Team – This is a library of attack technique implementations easily importable into the Purple Team platform for simulation.
• Our own Threat Intelligence Team gathered intelligence from incidents we have assisted with. That allows us to execute Purple Team scenarios based on real incidents to ensure that the same incident will not happen twice.
• SRA Threat Simulation Indexes – Aggregated indexes for how threat groups operate in the health, finance and retail sectors.

Automate all the things

When we started, we designed scenarios and test plans that involved us implementing the procedures for test cases and executing them manually. Initially, this meant we had hands-on keyboard testing performed over multiple days. This system worked, and we got the job done, but we quickly realised that executing test cases manually like this was far too time-consuming and redundant. Additionally, manually logging all our actions was a huge pain point, and one of our talented team members ended up with the unfortunate role of secretary for an entire test. We also saw that there was no reason that many of the one-liners we executed could not be automated.

Now, this is something we should have solved sooner. We tried addressing the "secretary" issue by more rigorously planning every test we were to execute. However, more was needed, as transferring what we typed on the keyboard to an Excel sheet or similar still involved too much overhead. We, therefore, started gradually using Vectr, but without automating any of the execution. That means we used it to "start" and "stop" exercises, which gives them a best-effort timestamp and log entry. This system also worked, but it did not sufficiently alleviate our issues, as this lacked precision and was prone to human error, e.g. "Did I press play before I ran that command?"

Now, at this point, we discovered that Vectr had an "Execution Framework", which enabled us to start automating test cases and entire campaigns. In short, it allows you to define a set of commands for each test case and build a binary or script that contains all the test cases you want to execute. Run the executable on the target machine, which outputs a log file imported into Vectr. This file then populates the time stamps for each test case and marks them as "Completed" by the Red Team.

This style of Purple Team testing solved nearly all our challenges with time-consuming planning and execution, as this was now largely automated. The accuracy significantly improved as timestamps were synced with the action executed, and the source and destination were largely prepopulated and easily trackable. We also discovered at this point that we got a ton of automation for free from Atomic Red Team, as it imports its implementations of attack techniques directly into the automation engine Vectr. Finally, the built-in reporting features of Vectr mean we are shaving hours of reporting and still producing much better reporting and tracking of results than we would have ever been able to do in Excel.

While this sounds fantastic, work still needs to be done. We still design many custom campaigns and test cases, but we now have a system we trust and think will last. Now, we spend a few days preparing an exercise instead of spending the same time onsite sitting in front of a keyboard testing. We're no longer holding up valuable IT security people for days, which has allowed us to scale the service and lower the entry barrier for the organisations we work with in a way we did not anticipate six months ago. So not only do we love and embrace automation wherever we can apply it, but in this case, it was necessary to continue delivering Purple Team exercises at scale to our customers.

What’s next?

By now, we've delivered multiple Purple Team exercises that have proven effective in advancing organisations' defences. Our Purple Team services help improve security detection controls and enhance response procedures and threat detection strategies. We're pleased with the organic growth of the service from just a general idea to something significant and easily approachable. We've learned a lot along the way and invested considerable time into professionalising Purple Team exercises to deliver them at scale to our customers. Here are some of the changes we are making right now:

Purple Team delivery model

We are ready to start delivering Continuous Purple Team testing, which allows organisations to identify and improve detection and response in real time and track progress over time. To support a continuous offering like this, we are launching a new tiered model for delivering well-researched and developed packages for Purple Team testing.

The first level will be called "Baseline security detection", and it will offer a starting point for evaluating existing detection capabilities and getting quick win improvements in areas such as endpoints, Active Directory and networks.

The second level, "Comprehensive security detection", supports more complex options to improve detection and response, including a ransomware readiness assessment. In the third level, "Advanced security detection", we will introduce tailored exercises like APT simulations and sector-oriented tests based on our threat intelligence.

The mnemonic Purple Team platform provides a visual overview of the planning, execution, reporting and tracking of these deliveries over time. Automation enables us to repeat exercises easily, show organisations how easy it is to start with Purple Team, and rapidly improve their detection and response.

Involving multiple discipline areas

As discussed in this blog post, integrating the Blue Team is critical to a successful Purple Team exercise. mnemonic is in a unique position where we often end up being both the Red and Blue Team on behalf of our customers. We deliver Purple Team as a Red Team-driven service with our Technical Risk Services (TRS) department. However, we're now gradually involving several of our other discipline areas in different ways, through for instane developing threat intelligence-based test cases and scenarios, building resilient detection and response procedures, improving and creating detailed detection signatures, providing strategic recommendations around detection and response in the context of business risk, and integrating our Argus platform and our MDR services with our Purple Team exercises.

Purple Team and MDR

In addition to everyone involved, we are integrating our 24/7 threat detection and response service Managed Detection and Response (MDR), with Purple Team. This combination has proven to be a success. We continuously improve the MDR service by conducting Purple Team exercises in mnemonic with MDR as the core detection mechanism. This approach benefits our customers and ourselves, making us all more resilient to attacks. Overall, this makes Purple Team and MDR a winning combination.

Want to learn more?

Read our Purple team Part 1 blog post, check out our Solution Brief for Purple Team exercises or contact us for a live demo.