Active Network Scanning in OT Environments
Periodic active network scanning is generally essential to maintaining an accurate picture of the network, as significant information is only available upon request and otherwise never present in normal traffic.
Passive traffic analysis is needed as many industrial control systems (ICS) devices were only designed to function as expected and are often not tested to maintain function when they receive traffic other designed. For example, I’ve seen RFID controllers and Anybus modules lock up, simply by receiving packets that confused them.
Active network scanning can deliver much more information than passive scanning, and can be an incredibly valuable tool in any industrial environment. However, as I noted in a previous blog post, devices in the industrial environment–including VFDs, PLCs, I/O blocks, actuators and sensors–can be more sensitive than those in the office environment.
Standard IT methods for network scanning cannot be used in an industrial environment without planning and forethought.
Precautions need to be taken; for example, scanning should be done delicately and while machines are not operating due to the potential that the added traffic could add latencies and other issues. Yes, some machines could run just fine during an active scan, but it requires an additional study to verify that.
Operators can get all of the benefits of an active scan without concerns for their network by incorporating the proper proactive, responsible and knowledgeable planning before performing an active scan. Usually.
I say “usually” because my colleagues and I were recently involved in a situation that did not go 100% as planned. However, it had a fascinating outcome.
Example: Active network scanning achieved through partnership
We were brought in to perform an active scan on a network operated by a large automotive manufacturer. This client is very sophisticated and we have a close partnership with them.
They knew just what they wanted–an active network scan that would quickly and efficiently go deep into their network to identify every device and provide extremely detailed information. Their goal was to receive rich data about all of their systems along with thorough analysis and recommendations.
Through their relationship with Belden, the client chose to work with Tripwiregiven proven experience and knowledge in this space. We scheduled the scan activity during a period that the line was down for scheduled maintenance. The scan was designed to be performed in a slow, gentle manner throughout the network.
We expected the impact on the network to be the equivalent of a light breeze. Yet, it was soon reported that a VFD tripped. Could the scan have caused this? Fortunately, all involved considered the situation and quickly concluded the scan could not–or should not–have caused the failure.
As it turns out, the source was an existing, hidden vulnerability in the VFD that could have been triggered by a multitude of disruptive situations, including a broadcast storm or series of malformed packets. It was incredibly fortunate that it was triggered harmlessly in this circumstance–if it had been triggered while the line was up, it could have been a serious issue, potentially shutting down production.
Our analysis confirmed for the client where the issue was stemming from and they approached the VFD manufacturer. To their credit, the manufacturer was grateful for the knowledge and agreed that what happened during our scanning activity should not have occurred. They tested the VFDs in their labs and addressed the issues, proactively correcting other reliability issues with modifications and firmware.
The end result is a better product and more reliable operation for all.
What would you do?
I'm sharing this situation given active network scanning still has a bad reputation due to situations where IT professionals have applied active monitoring methods common in office environments without adapting to the precision and often sensitive nature of the Operational Technology (OT) environment, causing adverse device interactions.
With this view may still linger, it may have been natural for the automotive network operator to assume the VFD tripped due to a poorly executed active scan. Fortunately, they were involved in and knowledgeable about the careful precautions taken to alleviate potential negative impacts while experiencing the benefits of active network scanning.
With open eyes, they investigated the situation and accurately identified the source of the issue. This was the initial step to resolution and ensuring it doesn’t happen again. I must also give credit to the drive manufacturer, who addressed the situation to further improve on a quality product. Taking the time to uncover the 'why' resulted in a strong partnership where everyone looked in the right direction and benefited from the learning.
About the Author
Zane Blomgren is Director, Industrial Cybersecurity at Belden. With over 20 years of cyber security experience, Zane has been called on to help build foundational security controls and assist after cyberattacks have begun. Zane marries his passion for security and his interest in collaborating with and learning from a wide variety of customers to apply best practices in even the most challenging of situations to create more reliable, secure systems for organizations in sectors including energy, transportation, manufacturing and many others.