CSWorks: web-based industrial automation

Of CSWorks and software development

Post-Stuxnet industrial automation systems

clock October 5, 2010 13:44 by author Sergey Sorokin

Introduction

Recently, I came across an excellent paper by Symantec engineers that sheds some light on what infamous W32.Stuxnet virus looks like from inside. First off, I was impressed. To be honest, I haven't had such interest reading about a dissected and prepared virus since mid-nineties when I was infatuated with assembly languages, and when viruses, worms and trojans just started making top spots in the newspapers. And here is why.

1. The scale of the research and development effort behind this attack is enormous. Simatic PLC programming, Step 7 internals, WinCC database vulnerability, Windows vulnerabilities (including two unreported exploits!), virus control centers in the web that parasite on some soccer site, compromised Verisign certificates issued to Realtek and JMicron, early versions of the virus that (presumably) conducted reconnaissance and gathered information about target hardware and software infrastructure... I can easily imagine big money spent on hardware and to keep busy a team of developers for many months. I can even imagine some black-hat hacker selling unknown Windows exploits and stolen private keys for the compromised certificates to this group - this type of product is marketable.

2. The determination of the aforementioned team to hit a specific target (or a group of targets). The detail level of the PLC part of Stuxnet gives the impression that someone with inside knowledge of how particular Siemens SCADA deployments work was involved. The effective use of system vulnerabilities and user habits is a trait of a very carefully planned action.

Anyways, after Stuxnet, SCADA world will never be the same. Let me share some thoughts.


Scapegoats? Anyone?

Iranian authorities have already detained several individuals presumably connected to the Stuxnet attacks. Being a peaceful tech person, I am absolutely not intrigued by the opportunity to see heads on sticks. Instead, I would rather take a closer look at the technical aspect of the story.

Siemens

By no means I am an expert in Siemens SCADA software, but to me it looks like WinCC database hard-coded password is the only serious security hole provided by Siemens in the Stuxnet case. Yes, using a hard-coded password that has been available on the Internet for several years cannot be considered a secure practice. But, after all, this vulnerability was used by Stuxnet only as one of the replication vehicles. Without this exploit, infection would spread out at slower rate, and that's it. As for the fact that a whole piece of Step 7 (Siemens PLC programming software) was replaced by the virus and executed to perform some malicious actions like tweaking PLC programs... Well, there is not too much Siemens developers can do if a virus gets privileged access to the system and can patch executable files in the memory and on hard and removable drives. This takes us to the next candidate.

Microsoft

These charges are harder to beat. According to the report, Stuxnet "exploits a total of four unpatched Microsoft vulnerabilities, two of which are previously mentioned vulnerabilities for self-replication and the other two are escalation of privilege vulnerabilities that have yet to be disclosed." This is big. These two unreported exploits gave the virus carte blanche to perform any memory and file modifications on all versions of Windows. Being a long-time Microsoft camper, I was happy to see the patch coming out on September 15, 2010.

IT infrastructure

While googling Stuxnet, I have come across this posting. Although the whole article is definitely worth attention, I would like to quote a couple of excerpts.

Question: Is it enough to separate the process control network from the business LAN? Is this realistic?
Answer: It depends on what you mean by “separate.” Industrial security standards and guidelines do recommend the networks be segmented by firewalls or other technology to prevent unauthorized communications between enterprise and control system networks. If by “separate” you mean “completely disconnected,” this is what control system folks mean when you hear the term “air-gapped.” In the pre-wireless days, the term meant “no electrons can pass between control networks and enterprise networks.”
Nowadays, very few control systems networks are completely disconnected from enterprise networks. Most control systems can still be run safely without enterprise network communications, but many physical processes can no longer be run profitably without routine, real-time enterprise network communications. ...

Question: We have found that changing the plant engineer’s way of thinking has been the biggest challenge with regard to increasing security on the control side.
Answer: I agree. There are many barriers to get past on the awareness side. Many plant engineers are focused on safety and availability objectives and see security measures as impairing their ability to meet those objectives. Others don’t believe there is a threat, or don’t believe their systems are sufficiently exposed to that threat. Many engineers simply aren’t aware – they don’t realize that their “air gaps” disappeared years ago.


To summarize all of the above, Stuxnet self-reproduction abilities would be extremely limited (and we would probably never hear of this attack) if the following conditions were met:
- all involved Windows system must be patched for known vulnerabilities (BID 41732, BID 43073, BID 31874);
- no public network shares;
- no removable devices (USB flash drives, CD-Rs);
- no WinCC database hard-coded password (I am not sure if this is achievable though);
- Step 7 stations should not have access to the Internet (no possibility to communicate to virus control centers);
- process control network (with PLCs and Step 7 stations) must be separated from other local networks and Internet.

Moving towards more secure SCADA systems

In the post-Stuxnet world, the term "SCADA virus" was coined by journalists and bloggers. Now, when I am introduced to non-technical people, I may hear: "Oh, you work with SCADA systems? You must be a virus expert!"  I find the term "SCADA virus" confusing and misleading and I tend to avoid it, and here is why. Stuxnet affects only one type of SCADA system and it doesn't really use SCADA for propagation - infection through compromised WinCC database is performed using OLE Automation stored procedures, which is a Microsoft SQL Server feature not related to SCADA. Siemens SCADA system just happened to be the target of the malicious payload part of the virus, but in theory it could be anything - a document management system (to get access to sensitive corporate materials), a CRM system (to borrow competitor's contacts and leads), an online game (to steal other players passwords).

The fact that Stuxnet is actually a "regular" virus is good news. This means anti-virus companies don't need to invent some special kind of defense and we can keep taking usual precautions to avoid infection. Just take them seriously - the stakes are a bit higher than in the stolen game password scenario. We already have all tools, we are aware of all good practices - let's just apply them properly. No user software running under super-user accounts, no removable media, firewalls, no weak/default passwords, no security by obscurity, controlled network access, code signing, up-to-date operating systems - the list is long but well-known.

I would like to elaborate the process control network separation aspect. Process control and PLC programming tasks can be sand-boxed using a dedicated local network. But HMI (Human-Machine Interface) applications that read and write from/to the controlling device require direct access to the device in most cases, intermediate database is not an option due to excessive latency. I can see only one solution here: HMI/SCADA developers must explicitly enforce production data access restrictions using standard, well-known and proven security solutions and practices. Every data read or write request from HMI client stations to PLCs must be authorized. No home-made security algorithms. No backdoors. No plain passwords sent across the network.

Security in web-based solutions

Let's get straight to the big question: are web-based systems more prone to cyber-attacks? Provided the server is able to properly authenticate the user and client-server communication channel guarantees data privacy and integrity, the answer is "no". As an HMI/SCADA developer, do I need to come up with some sophisticated algorithms to ensure proper user authentication and channel security? No. Internet security industry is sufficiently mature to provide developers with correspondent technologies.

There is also a psychological aspect that influences developer's behavior. When working on a web-based solution, developers constantly keep in mind the fact that that every piece of data sent between client and server is (potentially) constantly being watched by people with bad intentions. This thought fuels their sense of alarm and, eventually, makes them produce secure design and code.

"...only real security is a pair of wire cutters…"

I took this quote from this posting and it makes sense to me. Together with the simple diagram on the right, it explains why we probably will never see a 100% secure industrial automation system. Stuxnet was a loud wake-up call for SCADA developers and users, hopefully it will create additional motivation to move towards the "Security" vertex of the pyramid remembering the closing line of the Symantec paper:

"...Despite the exciting challenge in reverse engineering Stuxnet and understanding its purpose, Stuxnet is the type of threat we hope to never see again.”

References

W32.Stuxnet Dossier, Nicolas Falliere, Liam O Murchu, and Eric Chien, Symantec, Sep 2010
http://www.symantec.com/content/en/us/enterprise/media/security_response/whitepapers/w32_stuxnet_dossier.pdf

SCADA Watch: Things You Probably Wish You Didn’t Know, Paul Ferguson, Trend Micro, Jan 2008
http://blog.trendmicro.com/scada-watch-things-you-probably-wished-you-didnt-know/

Stuxnet Webinar Attendee Questions, Andrew Ginter, Industrial Defender, Aug 2010
http://findingsfromthefield.com/?p=510



Security - access rights management

clock September 7, 2010 18:06 by author Sergey Sorokin

CSWorks provides a mechanism to support LiveData, Alarming and Historical data access authorization, but does not have access rights management functionality. Companies that deploy web-based solutions usually have some access management infrastructure in place, which is either based on Active Directory or integrates with some third-party technologies. And in most cases, IT managers in these companies do not like the idea of having yet another piece of security infrastructure that can be used for SCADA-related authentication tasks only. It's just too much hassle: extra support, one more set of user names and passwords to maintain, extra security risks.

This is why we do not offer our own access rights management subsystem with CSWorks - we want to let IT managers choose the right one. We designed CSWorks in a way it can be integrated with existing security solutions. It's up to the application developer team how CSWorks-powered ASP.NET application authenticates a user and what screens are available to the authenticated user. CSWorks can only enforce some restrictions on data access when a CSWorks-powered client application accesses data - LiveData Alarming, or Historical (see CSWorks documentation for details).

If you are new to the world of the web-based access rights management systems, you probably need something to start with. I would recommend downloading a few products available on the market and getting familiar with them.
PortSight SecureAccess and Atlassian Crowd are good candidates: they both support Active Directory integration, both are .NET friendly, both are mature products, both provide samples for .NET developers. If you decide to use such a product for your CSWorks solution, your application should perform the following tasks.

  1. User authentication. Performed by the selected access right management solution either through API or through a separate web page.
  2. CSWorks web service access authorization. Authorization information is stored in the ASP.NET session state, see CSWorks documentation for details.
  3. ASP.NET page access authorization. Your ASP.NET application should provides access only to the pages that are available to the authenticated user. This can be done by using access right management solution API (direct .NET calls or web service-based).
  4. Silverlight page access authorization. Your Silverlight client application should provides access only to the pages that are available to the authenticated user. This can be done by a variety of methods: web service calls from the client applications or passing startup parameters to the Silverlight engine when starting client application.

Of course, there is a learning curve involved in some cases. Yes, third-party (or custom developed) software is involved. The benefits outweigh the risks though - customers get a robust solution that works in a predictable way and re-uses existing security infrastructure.



CSWorks Light - no IIS required

clock September 2, 2010 20:08 by author Sergey Sorokin

Some people who are willing to try CSWorks have no possibility or desire to install IIS (Internet Information Services). Starting today, to make their lives a bit easier, we offer a special distribution of CSWorks called "CSWorks Light" that does not require IIS. A couple of highlights:

  • this distribution uses Microsoft Cassini web server - a very simple and limited-functionality, lightweight alternative to IIS;
  • this distribution misses some samples that work with IIS-hosted CSWorks;
  • this distribution should not be used in production environment, it is for demo purposes only.

Cassini-based deployment has the following limitations:

  • it can host only one ASP.NET application per port;
  • it does not support HTTPS;
  • it does not support authentication;
  • it responds only to localhost requests.

Cassini was designed as a simple tool for debugging .NET applications and it is not officially supported by Microsoft (read full story here), so please do not expect stellar performance and production-grade reliability from CSWorks Light.

The link to CSWorks Light download will be provided in the email you will receive after submitting our
download form.



CSWorks 1.4.3880.0 released

clock August 18, 2010 20:28 by author Sergey Sorokin

What's new:

  • M2M Demo application
  • Shared data (databases and XML) in ProgramData folder
  • Setup: advertised shortcuts


CSWorks 1.4.3860.0 released

clock July 27, 2010 14:26 by author Sergey Sorokin

What's new:

  • Oracle support tested with Oracle 11g R2, Devart dotConnect for Oracle 5.70.146, Oracle.DataAccess.Client 2.112.1.0 
  • mySQL support, tested with mySQL 5.1.48, Devart dotConnect for mySQL 5.80.146 
  • Improved type control in LiveData stack 
  • Minor fixes in Excel demo


Error: LiveData Web Service is not available

clock July 22, 2010 04:41 by author Sergey Sorokin

The remote server returned an error: NotFound.

In some rare cases, after installing CSWorks and running Pipes and Tanks Demo, you can see the following error:



This means one simple thing: ASP.NET cannot run LiveData Web Service code. There can be different reasons for this. The first thing you should do in this case is to get as much additional error information as possible. Check application event log - it may give you some clues. Also, try to get LiveData Web Service definition from your browser at http://localhost/CSWorksDemo/LiveDataWebService/Service.asmx.

The browser will probably give you some more details about the error. Consider the following example.
 


The problem in this particular case is that ASP.NET cannot write temporary binary it compiles for a specific page or web service. There can be several root causes for that, for example: error in the ASP.NET installation, changing ASP.NET  worker process account without modifying temp folder privileges etc. There is a lot of information about this issue in the net, here are some good sources:

http://forums.asp.net/p/1060279/1520411.aspx
http://weblogs.asp.net/rchartier/archive/2006/01/05/434626.aspx

The idea is to give the account ASP.NET runs under, say NETWORK SERVICE, or LocalSystem, or "ASP.NET v4.0 Classic" account (depends on which account you are using, see your IIS Manager settings, Application Pools, "ASP.NET v4.0 Classic" pool properties) full access to all temporary folder it may use while compiling ASP.NET page or service. Those folders include:

c:\WINDOWS\Microsoft.NET\...\Temporary ASP.NET Files
c:\windows\temp

One would assume that ASP.NET setup (or aspnet_regiis command executed manually) should do that, but it looks like there is no 100% guarantee. Fixing folder access manually seems to be the best option. Unfortunately, there is nothing CSWorks installer can do here - it's ASP.NET setup problem.

The remote server returned an error: NotFound. (again)

Another scenario, same symptoms: LiveData Web Service is not available. The remote server returned an error: NotFound.

Navigating to http://localhost/CSWorksDemo/LiveDataWebService/Service.asmx gives the following error on a 64-bit system:

It sounds like ASP.NET worker process did not expect 64-bit code in the "bin" folder, so it cannot execute web service code. Let's open server manager and check Classic 4.0 Application pool settings:

Set "Enable 32-Bit Applications" to "False" and it solves the problem.


[HttpWebRequest_WebException_RemoteServer]

Another kind of error is common: LiveData Web Service is not available. [HttpWebRequest_WebException_RemoteServer]



This is usually caused by CSWorksDemo virtual directory ASP.NET version mismatch: CSWorks binaries require version 4.0, but the virtual directory uses another version. If you are using XP or 2003, right-click CSWorksDemo in IIS management console, select "Properties" and select ASP.NET 4.0 in the "ASP.NET" tab. If you are using 2008 or Windows 7, make sure you have selected "ASP.NET v4.0 Classic" application pool in CSWorksDemo virtual directory basic settings.

Another possible reason for this problem is: ASP.NET 4.0 web service extension (Windows 2003) or ASP.NET 4.0 ISAPI extension (2008,Vista, W7) is disabled. Please follow the instructions at
http://www.controlsystemworks.com/iis.html and enable correspondent extension manually.



Trend data export to Excel

clock July 13, 2010 16:14 by author Sergey Sorokin

Starting from version 1.4.3850.0, CSWorks Trend control is capable of saving trend data to CSV (comma-separated value) files on the client machine. The following video gives the idea how the export process works and how trend data can be used in Excel. It's easy.

Watch the video in high resolution (1280x720) to see all details.



CSWorks now speaks: en,no,de,ru

clock July 12, 2010 20:10 by author Sergey Sorokin

Starting from version 1.4.3850, CSWorks Alarm Summary and Trend controls support two more languages: German and Russian:



Those are the screenshots of CSWorks demo applications included in CSWorks install package. Please note that some of CSWorks demo applications now support culture parameter that can be passed in the URL. If you want to run CSWorks demo application using one of the supported cultures just specify "?culture=<culture>" parameter in the address bar.



CSWorks 1.4.3850.0 released

clock July 6, 2010 14:38 by author Sergey Sorokin

What's new:

  • Client development: integration with Expression Blend 4
  • Improved downsampler engine now used on both client and server
  • Alarm Summary and Trend control in German and Russian
  • Trend Control: data export to comma-separated files
  • Minor OPC emulator fixes


Silverlight 4 - relative web service address support

clock June 25, 2010 16:59 by author Sergey Sorokin

Silverlight 4 supports the resolution of relative service addresses. Provided the Silverlight XAP package is hosted at http://somesite.com/application/ClientBin/Application.xap, the following address resolve as indicated:

"../Service1/Service.asmx" resolves to “http://somesite.com/application/Service1/Service.asmx”

For you, CSWorks users out there, it means one good thing: no more ServiceReferences.ClientConfig hassle, see Running CSWorks demo applications from remote clients post for details. Now you can simply specify relative service addres as follows:

<endpoint address="../LiveDataWebService/Service.asmx" ...>

and your CSWorks application will work from any website you deploy it to. Switching to Silverlight 4 pays off!