Reading Time: 10 minutes

Investigating an IDS Alert for Shell Code Heap Spray

An Example of Pivoting Through The Noise

It only takes a short amount of time as a Packet Nerd doing Network Security Monitoring to understand a large part of the job is eliminating false positives from the queue. Tuning out repeat offenders from alerts is one of the primary goals of any SIEM operator or analyst. Noise reduction is key in detection. Some alerts you don’t want to tune out because of the potential severity. Alerts for Shell Code are always worth looking into. Update packages from many vendors will often trigger Shell Code alerts based off of strings similar to heap sprays (as an example). Every alert (or group of them) with potentially high impact needs to be investigated. Such a demand requires analysts develop and utilize methods to quickly sort the good from the bad.

The following walk through is an example of such methodology. It was an alert caught by the home SIEM. In an unplanned turn of events the PCAP for this event had already been removed from the system due to limited disk space. However, as hackers we always strive to have another way to meet our objectives. The Security Onion scrapes and sorts a lot of metadata about the connections it sees. This enables us to quickly piece together the story even without the PCAPs available to us.

Abstract

An alert for Shellcode was logged by an Intrusion Detection System on a home network.  A key question must be answered to determine if it was a false positive or not, “Who/where did the requested data in the connection come from?”. Answering this will quickly determine to a reasonable degree if it was a benign event. In this case there were no other corresponding alerts or signs of malicious activity (such as callbacks to C2 (Command and Control) servers) which allows for this alert to be quickly answered without the need for reverse engineering or deep analysis of the files involved. The PCAPs for the alert are not available and metadata is used to determine the connection was between a host and a CDN. The Zeek logs of observed DNS transactions were searched and it was quickly determined the destination beyond the CDN was a Games Launcher. Many alerts are False Positives and the ability in spotting such events is just as relevant as finding evil.

The Alert

Checking the alerts in Security Onion or any other SIEM is like checking crab pots: its always interesting to see what you got. The following walk through is centered around using a Security Onion. These same techniques can easily be adapted to be used with almost any product on the market. The intent is to share methodology – not clickology. That said, there will be tips on using the Security Onion.

Certain alerts always warrant investigation. One such group is anything with the word Shellcode in it. Shellcode is a small amount of code used in exploitation of vulnerabilities that allows the attacker to establish a connection with the victim machine. Two such alerts are seen below in the Security Onion Sguil Client.

Understanding an alert, or group of alerts, is key in triage. Analysts should start with the most serious alerts and quickly understand what might be related.

1. First any alert with ‘Exploit’, ‘Shellcode’ or ‘Suspicious’ should be looked at carefully. This is not an all inclusive list.
2. The internal host which was involved in the alert must be noted for context.
3. Do the external IP’s look familiar? Are they external?
4. When did these things happen? This will assist in quickly understanding the whole story.
5. Which sensor detected it? Was this in our crown jewel subnet? Note: Security Onion it is SensorNumber.AlertId.
6. How many instances of this alert occurred

A quick analysis is that an internal host exchanged packets with two different external hosts that triggered an alert for Shellcode nearly 50 times within a matter of moments.

Figuring Out the Who in the 5 W’s

Who’s Alert is it Anyway?

One of the first things that must be established is which piece of friendly infrastructure was involved. Knowing the role of the system involved will provide a lot of context to the investigation as progresses. Zeek captures DHCP transactions. As long as the sensor was in a position to observe these requests and their answers, investigators can quickly determine who the monitored host was and provide more context. In an Enterprise environment this is quickly done by recognizing the IP address.

Keep in mind DHCP transactions don’t occur for static servers and this will likely come into play for user workstations. It is absolutely key in any part of the investigation that timelines and timestamps are checked against the data. What the DHCP server assigned 2 days ago may not be what it assigned at the time of the alert. In this case the DHCP logs for the past 24 hours were checked to ensure the proper host was identified.

In this case it appears the monitored host involved is a computer named Drogon.

Who Did the Host Talk To

Experience dictates this is likely a package of updates with 0x0a repeating in the data. This theory must be confirmed to resolve the alert. A quick win to determine if this was the case is to understand what domain the host reached out to for this data. Checking the PCAP, or the transcript of the connection, will quickly reveal the answer.

PCAP storage is expensive. Many systems trim the PCAP data by replacing the oldest data as space is needed. This also known as “rollover”. It is dependent on the amount of space available to the SIEM for storage. A large amount of data was recently downloaded and the PCAP data for this alert was gone less than 24 hours later. The Security Onion had nothing to display for the transcript or PCAPs. This is acceptable on a home network lab (where this was captured); however in an Enterprise environment there is likely a need for much longer retention of PCAP data. Despite the cost full packet captures can be priceless in investigations.

Reverse engineering the file is arguably the only way to be sure this alleged heap spray was benign. In this instance the file is not available, and reverse engineering every file flagged like this would overwhelm any SOC and its analysts.

Metadata to the Rescue

The Security Onion uses Zeek to carve and sort through every connection observed. As Zeek examines each connection it makes a lot of determinations about the connections and logs them. Security Onion in turn gathers these logs, sorts them and displays them in Kibana for easy analysis. In this case the metadata is enough data to make reasonably confident resolution on this alert.

VirusTotal

Pivoting from the alert to VirusTotal is a quick way to begin understanding what domain the host was attempting to talk to at that IP Address. Right clicking the Source IP Address will display a feature rich context menu for pivoting.

Transitioning to the VirusTotal page for this indicator reveals no known detections. This is a good start but still does not fully answer the 5W’s that every analyst must strive for. Side note: no detections does not mean it was not malicious! It merely means it isn’t involved with any interesting findings yet.

Note that Akamai is the ‘owner’ of this IP address according to Whois and VirusTotal. This is a common dead end for those unfortunate enough to not have access to DNS logs (you are logging those right?) or the metadata of the DNS connections.

To summarize what is happening here: we have an IP Address as an indicator involved with an alert. The goal is determine what, or who, the host was reaching out to when this alert triggered. An IPv4 address is no longer enough to determine the answer. In many cases the IPv4 address simply resolves to a CDN (Content Distribution Network) like Akamai. CDN’s are systems distributed throughout the world to deliver content for many different customers. This means when a trail leads to a CDN it is blind spot and dead end. To get around this we have to determine who the host was asking for when the connection was made.

VirusTotal reveals historical data regarding DNS resolutions involved with that IP address. Simply select the ‘relations’ tab in VirusTotal.

VirusTotal tells us the most recent records logged were for Dell.com. A careful analyst would note that the alert in question was from July 22nd, 2020 and this latest record was from 3 months earlier in April. Being talented forensicators we know we must validate data, especially older data, with a second source if able. IPv4 Addresses and their domains can shift quickly in addition to a single domain hosting hundreds of domains or more – especially when a CDN is involved.

DNS Logs

When the sensor is in a position to observe and collect DNS queries and answers we can get a good understanding of where a host was going based on matching the IPv4 address with domain name resolutions requested by the host. Security Onion allows us to review logs of these transactions. To do this from the Kibana Dashboards select the DNS Dashboard and look for IPv4 indicator in question.

As discussed previously it is imperative that we are looking at data from the proper time frame. Since this alert happened in just under 24 hours prior to the investigation, “24 hours” was selected in Kibana. It can be seen here that there are 3 logs for the IPv4 address in question. Scrolling down the page to the Answers and Queries section will shed light onto where the host was going at the time of the alert.

Thankfully, the name of the domain requested appears to be a benign request for downloading data from Epic Games Launcher, and possibly the Origin Games Launcher. To isolate this data hover the mouse to the right of the count (1) and select the (+) magnifying glass. This will filter the data to only this domain. This filter can be removed at the top of the page (demonstrated later). This technique is a quick way to reduce ‘noise’ in the investigation and assist in visualizing and understanding the scope of the incident (as an example).

Conclusion for this Alert

It is highly probable this was an update, or series of updates, through the Epic Games Launcher. There could be an argument made that to be completely sure the files must be examined. However, based on the evidence, and lack of corresponding alerts, this appears to be a benign download and thus ruled a ‘False Positive’.

The Next Alert in the Cluster

There was another alert in the cluster of alerts for Shellcode that should be investigated. The techniques above were used and resulted in similar results. For brevity, only slightly different techniques for pivoting with the Security Onion are shared. These same techniques exist in other SIEMs and can easily be adapted. Understanding the pivot points in the data and where to use them is key.

In Security Onions Sguil client simply right click the source IP and conduct a Kibana Lookup.

This will open a Kibana Indicator dashboard for the indicator. Always double check the time frame involved!

The Indicator Dashboard quickly displays a wealth of information. The dashboard enumerates the data types involved with that indicator. An investigator can quickly ascertain that there 44 connections from the monitored network to that distant host, 35 of which were HTTP based; there were 27 files exchanged with that distant server; 19 alerts were generated with that server in the time frame selected; and finally 3 DNS transactions were logged. In the NIDs Alerts it is noted that all 19 alerts were for Shellcode. This is KEY! The lack of related alerts indicates that this was not a successful attack – or more likely is a ‘False Positive’.

If this was successful we would expect more alerts to this server. It must be noted that many malicious actors will not have the initial download served from the same location as the second stage, or the follow on Command and Control (C2) connections. To further the investigation it would be wise to pivot back off of the monitored host to see if it had other alerts from around the same time (or after) the exploit allegedly seen here. It is not done here because of the answer to the question, “Who was this host talking to?”.

Should filters be applied through the course of pivoting through the data there will be a point were removing all of the filters will come in handy. To do this select the action menu at the top of the page (1). Next select ‘Remove’ to remove all filters (2).

Using the Discover Section of Kibana

This is similar to sifting through data with Splunk searches etc.

Creating effective queries to sift through millions of logs is key and probably deserves a post in itself. However, a quick example here would be a search for files seen by the sensor where the indicator in question was involved. In this case the query would be:

event_type:bro_files AND "23.46.30.59"

Data from the results show a hash was taken of the file. The fields the actual hash can be found further down the page.

VirusTotal and other threat intel databases can be used to cross reference hashes of suspect files discovered during the investigation. A search for this hash in VirusTotal returns 0 results. A hash, or file, with zero results simply means VirusTotal has not seen the exact instance of this file. Unknown hashes indicative of malware not previously seen (perhaps uniquely generated for this attack); or likely as the case in this instance it is simply an update file that no one has sent to VirusTotal for examination.

Conclusion

Investigations can be messy. All the desired data sources investigators hope for may not be available. Forensicators must answer the 5 W’s using what they have available. PCAPs were not available in this case, but the metadata still created a clear picture. Lastly, the investigation would have stopped short of a satisfactory answer had we simply said the file came from Akamai (a CDN). By pivoting to host DNS requests we were able to conclude with decent certainty that the connections which appeared to be shellcode on the wire were in fact benign downloads.

Pivot Recap:

Who was the internal host: Host Alert > DHCP Logs to confirm which internal host

Who was the external hosts: External Host IP > CDN Domain (Dead End) > DNS Requests seen from the internal hosts revealed it was game updates.

Enjoy the thrill of the hunt and keep at it until the every means to draw the story have been exhausted.

Happy Hunting.