Why SCADA Firewalls Need to be Stateful

What is a stateless firewall?

Following on from Eric Byres’ discussion of Deep Packet Inspection (DPI), this article discusses a second and equally important aspect of effective firewall security referred to as “stateful inspection.”

In order to understand exactly what is meant when we talk about “state”, we need to look at the specifics behind the TCP communication sessions that are most common in modern day industrial control systems (ICS) and SCADA applications. The figure below illustrates the model.

Figure 1: The abstraction layers typically used in TCP/IP communications for ICS and SCADA systems.

The DPI that was previously discussed is actually analyzing and making decisions based on the information contained in the upper layer of the model. This layer is where you would typically see specific application operands such as a Modbus Read Coil (e.g. function code 2) or a Modbus Write Single Register (e.g. function code 6).

Clearly this DPI provides the highest level of granularity when it comes to managing communication between hosts on a network. An example of an industrial firewall that offers this is the Tofino Modbus TCP Enforcer.

It is also possible for appliances to offer what is called “Shallow Packet Inspection” or SPI, which looks at data lower in the protocol stack. This is where the concept of “state” becomes important.

State in Data Communications

Just like communications between people, communications on a network rarely just involves a single message. There is a constant exchange of packets, where each is in some way dependent on previous packets. For example, if your computer receives a message containing a web page from a web server, it can only be interpreted with the knowledge of what request was sent earlier. If your computer never sent a message asking for the page, then the only reasonable thing to do with the message is to discard it.

This understanding of the previous traffic is what we call “state”. Basically the computers involved in the information exchange have to understand the “state” of the exchange at all times. They need to know state information such as which device started the session, who last sent a message, was the last message rejected because of errors and so on. Without this, communications quickly breaks down. 

The Hazards of Stateless Security

Compared to the devices actually communicating, a security device has a little more flexibility regarding understanding the state of the traffic it is securing. In theory, it can try to manage the traffic while ignoring the state of the sessions on the network. Unfortunately, ignoring state comes at a price – badly degraded security. Let’s explore why this is so.

A “stateless” or “packet filter “appliance only analyzes each packet in isolation of other information relating to the communication session. It has a series of static rules and uses them to take action upon received packets on an individual basis. It bases decisions to allow or deny packets on simple filter criteria such as:

• the source and destination IP addresses in the message (Layer 3)
• the source and destination protocol number in the message (first part of Layer 4).
  
  

The following rules can easily be handled by a stateless packet filter firewall:

• Accept Domain Name Service (DNS) response packets on User Datagram Protocol (UDP) port 53;
• Block any traffic to or from Internet Protocol (IP) address 24.116.25.21;
• Prevent web surfing by blocking outgoing packets from accessing Transmission Control Protocol (TCP) ports 80, 443, 3128, 8000 and 8080, unless they are directed to the corporate intranet web server's IP address.

Unfortunately, a packet filter firewall’s inability to understand the relationships between a series of packets is a serious weakness. For example, the broad rule "Accept DNS response packets on UDP port 53" contains a serious flaw. What if no DNS query was ever issued, but a spoofed DNS "response" packet came in instead? This simplistic firewall would accept the packet and deliver it to the "requesting" host, possibly causing it to become confused.

Hacking a stateless firewall

In the first part of this post, I explained what a stateless firewall is and the hazards of stateless security. Below, I will show you just how dangerously insecure these devices are.

Setting Up the Stateless Firewall

Let's consider a simple session where a client computer issues a request to a web server using the HTTP protocol as shown in the figure below. As defined in the IETF specifications, this message will contain the IP addresses of both computers ("src.ip" and "dst.ip"in Figure 1). It will also contain the number 80 in the destination port ("dst.port") field to indicate that the TCP packet contains a message for a HTTP server.

The HTTP server will reply with a message that has the "src.ip" and "dst.ip" reversed (since the message is in the reverse direction). The number"80" is moved to the source port field, indicating a reply from a HTTP server.

For completeness, this first illustration includes what is called the "TCP 3-way Handshake" shown in yellow in Figure 1. This handshake is required to establish communication for all TCP communications and is also part of the "state". However for today, we are going to skip the details of the 3-way handshake and focus on what happens after that.

Figure 1: The TCP 3-Way Handshake and subsequent communications.

Now to manage these communications, let's insert a simple switch or firewall that is unaware of the "state" of the communications. In other words, the device is unable to determine anything about any previous communication that may have taken place between the two hosts (including the presence or absence of the TCP 3-way handshake). The only information that this appliance can inspect are the IP addresses (source and destination) and the protocol port number (source and destination).

Now in order to allow our simple HTTP session to pass through this appliance, we would need to configure a unique rule for each "direction" of the session - one to allow the outbound HTTP "request" and the other to allow the inbound "response". These rules might look something like this:

Allow src.ip=10.1.1.100   dst.ip=10.1.1.200   dst.prt=80 (Outgoing rule) 
Allow src.ip=10.1.1.200   dst.ip=10.1.1.100   src.prt=80 (Incoming rule)

Spoofing Stateless Communication is Easy

While this looks simple, this type of paired access control rules immediately leads to security vulnerabilities. Since the appliance can only analyze limited source and destination data, it is not possible for it to block "inbound" communication that was not the direct result of an "outbound" request!

To exploit this hole, an attacker could simply change their IP address to match the Application Server (this is known as "spoofing") and send their malicious packets to the Client marked with TCP port 80 as his source port. (Note: this attack allows the packets into the control system - if you want to get replies back from the client you would require a second technique called arp poisoning, but that is another discussion).

Figure 2: Spoofing is possible when an attacker sends malicious packets using TCP port 80.

Not only is this simple to perform, but in this case, it is also difficult to detect since the traffic would appear as normal web browser traffic to monitoring tools. The stateless firewall or switch would only see the traffic as coming from the correct IP Address and as being some sort of HTTP message, and happily let it through.

Now this is a moderately serious security problem if you have configured your stateless firewall to only allow web traffic to a single server; at least that forces the hacker to pretend to be that one server. But what if you want your user to be able to browse the web?

The resulting stateless rule would allow the entire web to be able to send any packet they want to the poor client computer, provided they mark the packet with source port 80 to indicate HTTP traffic. It won't matter if it is really HTTP traffic – it would just have to be marked as such. That is a very low bar to jump over. Plus the hacker can send as many of these packets as they want, a wonderful way to flood the control network.

Recap of Stateless Security

1. A security appliance that is Stateless can only analyze each packet using limited source and destination data. It is unable to determine anything about any previous communication that may have taken place between the two hosts.
   
   
2. It cannot block inbound traffic that was not the result of an outbound request.
   
   
3. Attackers can send malicious packets by using an IP address that matches the Application Server. This is known as Spoofing.
   
   
4. Spoofing is difficult to detect as it appears as normal traffic to monitoring tools. The volume of traffic is higher, but this will probably not be noticed.
   
   

Stateful Inspection and Securing ICS and SCADA Systems 

In the beginning of this post I explained what "state" means in network communications and the hazards of stateless security. Then I detailed the behavior of a stateless firewall and included a demonstration of me attacking one. In closing this post, I will describe stateful inspection and its importance in securing ICS and SCADA systems.

Stateful Firewall Inspection

Stateful Inspection occurs when a security appliance understands the relationship between the latest message received and previous messages in a given network connection.

At its most basic, the stateful security device knows if an inbound "reply" packet is in response to an outbound request. But stateful inspection can be more than just that. 

Stateful filtering requires that the appliance look at additional parameters contained within the TCP/IP Headers. This additional information will indicate the state of the communication session as the device sending the message believes it to be. Typically this includes:

• Source and Destination TCP port number
• IP Status flags
• Sequence Number
• Acknowledgement Number

At the same time, the security appliance also creates an internal record that tracks its understanding of the session state. Together the state information as received, and the state information as previously recorded, can be used to analyze the reasonableness of each packet.

Inbound Traffic must Match Outbound Requests

Stateful Inspection means that inbound traffic to the client device will only be allowed if it is in response to an outbound request. All other traffic, even if it contains the correct IP address and TCP port number would be dropped. With "Stateful" inspection, an attacker that attempts to spoof a valid host's IP address and the TCP port number used would not be able to traverse the firewall appliance.

PLC Security Example

In the earlier blogs, we looked at the issues of traffic going out of the control network to an external web server. With a stateless firewall it was pretty simple to blast fake reply messages back into the control network.

But let's take the same protocol, namely HTTP, and look at a different example. As followers of this blog have heard, many PLCs have a small embedded web server that allows a user to point their browser to the PLC to make configuration changes. The user makes an HTTP request, using their browser, to the PLC to view the configuration options available to them.

Behind the scenes the browser has made an HTTP request (using TCP on port 80) to the PLC. In doing so the browser has made the initial 'request' to that PLC, with the PLC making the initial 'reply' to that request. This is a matching sequence of events. How would a stateless firewall versus a stateful firewall handle this transaction?

In the case of a stateless firewall, the sequence of events is irrelevant. As long as the data is TCP and sent to port 80, it would be allowed through the firewall, regardless of what sequence of events happened before.

This makes it extremely easy for a hacker to execute a denial-of-service attack against the PLC. A hacker can simply send the exact same TCP packet with destination port 80 thousands of times and the firewall will permit this traffic through.

It does not take into account any previous packets sent. The traffic will overwhelm the PLC and the stateless firewall is unable to prevent the attack.

A stateful firewall would map each request and reply to a state in which this is valid according to how this communication sequence should behave. It builds an internal record of each step of the communication so that if a hacker were to send random out-of-sequence data to TCP port 80, it would drop all data not adhering to the proper sequence of events.

The stateful firewall also has a maximum limit of the number of new TCP sessions allowed, thus protecting the PLC from becoming overwhelmed with connections.

In summary, the stateful firewall will block the out-of-sequence packets and permit only a maximum number of new connections, therefore stopping a hacker in their tracks.

Designing Secure Industrial Control Systems

Considering what you have learnt about "state", I would like to offer one design aspect for your consideration. What do you think is important when you are selecting a security appliance to protect a high-risk device such as a PLC that is connected directly to your manufacturing process?

In previous blog articles we have learnt about the importance of using deep packet inspection to manage the actual content of the traffic sent to a PLC. This prevents unauthorized commands from being maliciously or accidentally sent to the PLC from otherwise authorized computers. However, when would state be equally important?

If all of the equipment is installed on a local network that offers strong physical security, there is lower risk that an attacker could successfully originate an attack from the "Trusted" PLC side of the security appliance.

However, consider this same PLC installed at the end of a wide area network connection, a situation that is quite typical in distributed SCADA architectures. In this case, extra attention should be given to making sure that a physical breach of a remote location does not allow unrestricted access to the internal control networks.

The Relevance of Stateful Firewalls in Today's ICS

Why does the modern controls engineer need to understand state and know when to demand it? First, it is important to understand the concepts of "stateless" and "stateful" and be able to assess the importance of stateful inspection given the risk mitigation desired.

Next, do not assume that a vendor's firewall or security appliance is stateful. While many are, there are significant differences amongst suppliers. Remember that we are talking about the firewalls that are used to protect the highest risk component in the ICS architecture – the control devices. These devices are physically connected to the process under control, and as such, have a different security profile than is required in other areas. Our experience shows that industrial field firewalls (that are typically fanless, industrialized for harsh environments, operate on low-voltage CD and are ICS protocol friendly) vary greatly in how they handle state.

In addition, do not make the mistake of thinking that a layer 2 switch with ACL rules is a firewall. It is not. It will not block out-of-sequence packets, nor prevent denial-of-service attacks to a PLC.

No "One Size Fits All"

There are always trade-offs between performance, security, ease of use and ease of maintenance that must be considered when considering the appropriate security appliance. However, what is most important is to understand the advantages of each type of firewall, and the specific risk mitigation they provide when designing a particular architecture.

In ICS / SCADA security, there is no "one size fits all". Each solution should be based on a consistent risk-based approach to threat management.

[1] the model discussed in this article is a simplification of the OSI 7-Layer Model