What you learned in school - applied to Sierra Nevada Brewery

I remember some discussion in my recent IT classes about the "network effect", an idea that academia uses to describe a good or service whose value increases as more people/nodes have access to it. For example, a technologically more advanced social networking site would likely run into a brick wall against Facebook due to their existing customer base. 

It took me a bit to put two and two together to realize that this same "network effect" was the driver in the Sierra Nevada Brewery case study video that I recently commented on. In this case, the effect applies to access to accurate, timely, information within the organization. For example, the QA department benefits from the fact that Fermentation and Filtration use the same system, even for their own purposes.  

This case study highlighted the value of obtaining IT support, whereas the norm for industrial users tends to be a struggle. The other point was enterprise wide workflow integration, where their initial pilot system was deliberately created in two areas that couldn't slow production - data acquisition and reporting. 

It's cool to see when an organization can access its data centrally and apply that to the business flow across departments. This is a microcosm of what I understand is a significant aspect of what the Digital/Internet Revolution is supposed to bring.  

Ignition paving the way with alarming - 7.6 public Beta released

The public Ignition 7.6 beta was just released. It adds a bit of complexity and brings a whole lot of capability. The main feature set is an alarming revamp that I believe pushes the envelope for automated SCADA notification. I'm not sure how many of the individual features are "firsts", but I suspect a few. For example, SMS text messaging and email both support 1 or 2 way communication, meaning you can acknowledge alarms by replying to a text message or email. Voice enunciation is supported via the SIP protocol, giving you the option of using a commercial Skype service, a cheap "asterisk" appliance box plugged into a phone line, or your existing commercial VoIP setup. The old "Intel/Dialogix/Win 911" solution I used with RSView years ago for a small hydroelectric plant did this enunciation in a cool way, but was awful for numerous reasons. The Ignition system runs everything through the "store and forward" system, which caches all historical information if connection to the SQL database is lost. It also stores all the alarm logs, system logs, and historical data in a consistent format so you can leverage pre-canned widget viewer objects. There are lots of different deadband/delay/grouping options to manage flapping values on the border or simultaneous sets of alarms. Flexible schedules and rosters are available. Most interesting, is the notion of "Alarm Pipelines", which are graphical representations of complex alarm logic. You drag and drop a flowchart to create a potentially complex system of: notifications, expressions, set properties, delays, and jumps to other alarm pipelines.

Some of the Ignition beta documentation:

Ignition 7.6 - Alarming - Beta (In progress)

Ignition 7.6 introduces a completely redesigned, feature rich alarming and notification system aimed at tackling all of your complex SCADA alarming needs.  These new features are the culmination of many years of customer feedback, as well as in depth research into SCADA alarming best practices and should put Ignition well in the lead industry-wide for alarming capabilities.

Overview

System-wide changes, from the Ignition platform to the introduction of new modules, have taken place in order to best incorporate the new alarming features in a seamless fashion into the Ignition software ecosystem.  Each of these changes are addressed in detail throughout this document, but to begin with let us take quick, higher level look at the new architecture.  The hope is to gain a little familiarity with new terminology and a loose understanding of how all of the pieces are tied together before diving into detailed explanations of how everything is configured.

Pieces of the Whole

As you can see from the above architecture diagram the new alarming system can be divided up into three major areas (Ignition Platform, Vision Module, Alarm Notification Module) with each area being home to several pieces of the new alarming system.  Each of these pieces are interconnected and provide different pieces of functionality to the alarming system.  

Ignition Platform

1. Tag Configuration - You’ll notice right away when editing a SQLTag’s alarm configuration that this has changed dramatically in 7.6. Tags can contain multiple alarms. Each alarm’s properties can be configured to be dynamic via binding them to tags or expressions. Associated data can be added to an alarm, allowing for custom organizational structure or adding process data to be stored along with the alarm event. All previous alarm configuration will be automatically converted when upgrading to Ignition 7.6.

1. User Sources - This is where you configure users (both for authentication purposes and notification purposes) and roles in Ignition. If you’re a user of Ignition 7.5 or earlier, you’ll notice that this is an evolution of the Authentication Profile system. User sources are responsible for providing Ignition users along with their associated username, password, and contact information needed for notification.
2. Schedules - Schedules define day and time availabilities that you assign to users.  They are used for determining when a user is available to be notified of active alarms.
3. Alarm Journal - The new Alarm Journal system replaces the old alert log. This journal logs all alarm activity to an external SQL database. The table structure is managed by Ignition. Querying is mostly done through new scripting functions. Querying can also be done directly against the database if need be.

Alarm Notification Module

1. Pipelines - Pipelines are a new construct that define the logic between an alarm going active/clear and people being notified. They logic graphs and are configured in the Designer via a drag-and-drop UI. Pipelines are where you can add logic like delays, message consolidation, escalation, etc. Note that pipelines are the “heart” of the notification system - you need this feature to do any alarm notification.
2. Email Notification - Email notification is included in the Alarm Notification Module. Email now has the capability to support acknowledgement as well.
3. Upgrades/Licensing - Note that the alarm notification is a new module, but since it covers functionality that was previously in the platform, users who are upgrading from pre-7.6 who are covered by our upgrade protection will get this module comped as an “upgrade”. This module will retail for $1,995.

Vision Module

1. Alarm Status Table - The vision module has a newly renovated alarm status table. It is much more customizable than the old table: row styles are dynamic and configurable via a customizer, columns are configurable as well. It has an ad-hoc chart feature for tags using the SQLTag Historian. This is also where the shelving feature is accessed.
2. Alarm Journal Table - There is now a built-in table for viewing alarm history, now called the alarm journal. You’ll find this component on the Tables tab of the component palette. You’ll probably want to bind the start/end dates to something appropriate, like a date range component.
3. Upgrades / Licensing - The major version number of the Vision module has not changed, so this does not constitute an upgrade.

SMS Notification Module

1. SMS Notification -  This module adds the ability to notify users via SMS. This is done using an Airlink Raven XE from Sierra Wireless. Communication between Ignition and this cell modem are over ethernet.
2. Upgrades / Licensing - This is a brand new module, so upgrades do not apply. This module will retail for $595 and requires the Alarm Notification Module.

Voice Notification Module

1. Voice Notification - This module adds the ability to notify users via a phone call. It uses a TTS (text-to-speech) engine to turn alarm messages into audio, and uses VOIP (specifically SIP) to make phone calls. If you do not have a SIP gateway on premise, you may use a commercial Skype account (requires internet) or an Atcom IP01 FXO gateway which will convert SIP to a plain old telephone line. (http://www.atcom.cn/IP01.html)
2. Upgrades / Licensing -  This is a brand new module, so upgrades do not apply. This module will retail for $695 with one voice, additional voices $99, and requires the alarm notification module.

Voice Alarming

The new Phone Notification Module adds the ability to deliver alarm notifications to users via telephone, using any SIP compatible phone system. Messages are constructed in text, and are delivered through a high quality text-to-speech engine. The engine supports multiple voices and languages.

Core Features

1. Deliver voice calls through any SIP compatible phone system. No dedicated hardware required.
2. Messages generated by high-quality text to speech, and not a canned set of prerecorded files.
3. Supports multiple languages concurrently, based on user preference.
4. Allows users to acknowledge events.
5. Supports requiring a personal identification number for additional security.
6. Ties into the audit log to audit call events, successful message delivery, and user acknowledgements.
7. Supports throttled message delivery.

Using Skype

If you don’t have an existing VOIP system in place, using a hosted service is the quickest way to get going with the voice notification module. Skype offers SIP based service through their Skype Connect product. To get started, you must have a business account, which provides you access to the Skype Manager. From there, you can create a new Skype Connect channel. For more information, visit http://manager.skype.com

Once you have created a Skype Connect channel, configuration in the gateway is similar to that of any other PBX system. The host will be “sip.skype.com”, and the user name and password will be those provided by skype during the registration process.

Schedules and On-Call Rosters

Schedules and On-Call Rosters are new to Ignition 7.6 and allow you greater control over managing who should be notified and when. These two elements work together when it comes to sending out alarm notifications to users.  

Schedules

Schedules define when a user is available to be notified, and rosters create a grouping of users that should be notified when certain alarms pass through different notification blocks in your alarm pipelines (see the alarm pipelines section below for an in-depth explanation).  Each notification block in an alarm pipeline has an On-Call roster property that associates the block with a roster.  When an alarm passes through that block the alarming system begins to work its way through the members (users) of the specified roster and notifications are sent to the users whose schedules list them as being available.

Alarm Pipelines

Alarm pipelines are something new to Ignition 7.6.  No longer is the logic for when, how, and to whom notifications are sent contained in the gateway.  As detailed above, notification profiles themselves have been stripped entirely of any functionality that doesn't have to deal directly with the delivery and receipt of notifications and acknowledgement.  All notification logic is now designed/specified and edited in the designer by creating Alarm Pipelines.  There is a dedicated workspace in the designer for pipeline construction that allows you to create and manage any number of unique pipelines.

Pipelines aim to give the user an easy to use, drag and drop mechanism for creating complex notification scenarios that allow for the incorporation of key features that were previously unavailable. Alarm escalation, notification throttling, dynamic/custom properties for alarm events, all of these features are now made available through Alarm Pipelines enabling you to customize pipelines to fit all of your alarming and notification needs. Understanding the flow of alarm events through a pipeline and how exactly pipelines work is key to being able to fully exploit the potential of Alarm Pipelines.

Alarms are configured on Individual SQLTags and each alarm is tied to an Alarm Pipeline. When the condition for that alarm is met a new alarm event is generated and sent to the entry point of it’s associated pipeline.  The alarm event then moves through the different logical elements of the pipeline until it finally reaches a pipeline endpoint upon which it then exits the pipeline.  There can be multiple alarm events in a pipeline at any given point in time and you can view the current status of the pipeline from the Alarm Pipelines status screen in the Ignition Gateway.  Some pipeline objects let an alarm event exit the pipeline it is currently in and jump to another pipeline entrypoint allow alarm events to be redirected when certain conditions are met. Once the alarm event reaches an endpoint of a pipeline that does not redirect to another pipeline it drops out then that alarm event’s existence comes to an end. 

Pipeline Blocks

There are six types of pipeline blocks that are used to construct your pipelines and each are used to accomplish specific tasks: Switch, Jump, Expression, Delay, Notification, Set Property, and Splitter.

Alarm Properties

Main -

1. Name - A unique name for the alarm
2. Enabled - Enables or disables the alarm
3. Priority - Choose from five priorities, from Diagnostic to Critical, for the alarm
4. Active Pipeline - Sets the pipeline to use when the alarm goes active
5. Clear Pipeline - Sets the pipeline to use when the alarm clears
6. Timestamp Source - Whether the timestamp comes from the system or from the value
7. Display Path - Optional path that will be used for display.  Takes a string separated by “/”
8. Ack Mode - Auto: Alarm is automatically acknowledged when it clears; Manual: Alarm must be acknowledged by the user; Unused: The alarm will always be marked as acknowledged
9. Notes - Optional notes regarding the alarm
10. Ack Notes Required - When set to true operators will be required to enter some explanation when the alarm is acknowledged
11. Shelving Allowed - Whether or not the alarm can be shelved (snoozed)

Alarm Mode Settings

There are ten different alarm modes to choose from.  The mode you select will determine any additional Alarm Mode Settings properties to be displayed.  Below are the different modes and their relative additional properties:

1. Equal

1. Setpoint - In this mode when the tag value is equal to this setpoint value then the alarm will be considered active.

2. Not Equal

1. Setpoint - In this mode when the tag value is not equal to this setpoint value then the alarm will be considered active.

3. Above Setpoint

1. Setpoint - In this mode when the tag value is above this setpoint value then the alarm will be considered active
2. Inclusive - When true then alarm will be considered active when the tag value is equal to or greater the setpoint value

4. Below Setpoint

1. Setpoint - In this mode when the tag value is below this setpoint value then the alarm will be considered active
2. Inclusive - When true then alarm will be considered active when the tag value is equal to or less than the setpoint value

5. Between Setpoints

1. Low Setpoint/High Setpoint - These setpoints create a range that the tag value must fall within in order for the alarm to be considered active.
2. Low Inclusive/High Inclusive - When true then alarm will be considered active when the tag value is equal to or less than/greater than the respective setpoint value
3. Any Change - When true any change in tag value that falls within the range defined by the high and low setpoints will generate an alarm event that isimmediately cleared. Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

6. Outside Setpoints

1. This mode operates similar to the Between Setpoints alarm mode.  You configure a range using the low and high setpoints and when the tag value falls outside of this range then the alarm is considered active
2. Any Change - When true any change in tag value that falls within the range defined by the high and low setpoints will generate an alarm event that isimmediately cleared. Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

7. Out of Range

1. Generates an alarm event whenever the tag value falls outside of the Eng Limits defined in the tag’s numeric section

8. Bad Quality

1. Alarm events will be generated whenever the tag reports having a bad quality

9. Any Change

1. Alarm events are generated whenever the tag value changes.  These events are immediately cleared and if the Ack Mode is set to auto they will also be auto acknowledged.  Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

10. On Condition

1. Is Active - This is a bindable property allowing you to specify an expression that will evaluate to true or false and determine whether or not the alarm should be active.  You also have the ability to bind this property to another driving tag.

Deadbands and Time Delays

1. Deadband - A numeric deadband for the alarm condition.  This deadband is separate from and evaluated after the tag’s deadband.  An alarm event will be generated as soon as the tag value crosses the active threshold but will only clear after the tag value goes outside the active range by more than the deadband.
2. Active Delay (rising edge time deadband)- The time in seconds before the alarm will be considered truly active after the active condition evaluates to true.  If a tag value crosses the active threshold and then clears before the delay period elapses then an alarm event will not be generated.
3. Clear Delay (falling edge time deadband) - The amount of time in seconds before the alarm will be considered truly clear after the clear condition evaluates to true.  If a tag value goes into the cleared state but then becomes active again before the clear delay period elapses then the alarm will stay active.

Sierra Nevada Brewery MES, ERP Ignition Case Study - Awesome!

This 5 minute case study video illustrates how Sierra Nevada Brewing Company ended up using Ignition as their MES and ERP platform. "We use it for everything as our go-to product". They do most of their design work in house, constantly updating their "slightly unique" process.

It's interesting to see the evolution of the importance and use of the brewery's SCADA system. They started with a small pilot project to capture PLC data, basically just as a stand alone OPC data logger. Another department tried out that same server as an HMI. The video shows how the demand signal for Ignition flowed from department to department by word of mouth. "Once it proved itself we started adding". Everyone really liked it - they added their section and wanted access to the growing realtime database: the brewhouse, fermenters, the cellars, filtration, the lab, packaging, tracking trucks, Quality Assurance (QA). Even their IT department.

The Ignition system grew with fan-based requests. The more information was available the more users craved. Everyone wanted to add their devices, sensors, and part of the process.  Once users see what's possible, they want a lot more: reports, charts, graphs, realtime and historical information. Sierra Nevada beer is tracked from tank to tank, information like "yeast cell counts", "CO2 values", and "temperatures", recorded automatically, which represents the production phase that the batch of product is in. They used to walk around the facility to track their beer through whiteboards. Now all of their information is tracked in their SQL databases, accessible from any desktop computer. Pretty cool!

Ignition in many industries

Inductive Automation 48 second teaser video of the many industries that Ignition SCADA software is used.

Appliances, automotive, building material, chemical, electronics, energy, fibers & textile, food & beverage...the list goes on.

Ignition and SCADA in the semiconductor industry?

I was poking around the Ignition Module Marketplace and noticed the "SECS/GEM" driver module for their OPC-UA server. Curiosity got the best of me and I was soon reading about the Semiconductor Equipment Communications Standard and Generic Equipment Model. While I find it a bit curious that the semiconductor industry, specifically SEMI (Semiconductor Equipment and Materials International) is big enough, or has some set of semiconductor unique requirements that justify an industry proprietary protocol. I remember a story in college from an "Intel Fellow" who headed up the early Pentium IV project. He described a colleague applying chaos theory to the sequencing of these $50 Million "machines" that would treat silicon wafers for weeks. My assumption would have been that, had existing standard protocols and standards not been sufficient, that the scale of semiconductor operations justify custom code to be written. I had not considered the semiconductor industry as another SCADA application. Thinking back, I remember how I envisioned automated manufacturing before ever doing any integration work. A giant factory rapidly creates parts and assembles pieces. Everything whizzes rapidly through an automated assembly line without any user intervention. One or two workers walk the floors with hard hats and clipboards to watch the magic happen. Does this sound like your process? "Real" facilities are a bit different. People are involved in the process, often continually changing it. Equipment must be integrated to work together. Budgets and efficiency are more important than ever. Simply put, most organizations that are smaller than Walmart cannot remain competitive by hiring teams of professional programmers to support their operations. Why would the semiconductor business be any different? I realize that my lack of experience in the field led me to put it on a pedestal. That industry can benefit from the rapid development capability that Ignition provides and, apparently, an $895 SECS/GEM driver. That's an exciting thought - I love to see the extensibility of the Ignition platform in action. From their web site:


The SECS/GEM Module enables Ignition projects and third-party applications to communicate with semiconductor fab equipment. The module utilizes the HSMS-SS and SECS II standards to communicate with equipment over Ethernet. In the near future, SECS I for serial connections will be supported as well. The module achieves interoperability with third-party systems through the use of shared database tables. For use with Ignition software – a rapid application development and deployment platform – built-in mapping to tags or scripting functions can be utilized. Driver configuration is done with a web-based configuration interface which includes a built-in editor for creating custom SECS messages. Over 100 SECS message definitions from the SECS II and GEM standards are supported out of the box. Ignition and this driver are cross-platform capable because both were developed in Java. An equipment simulator is included with the module to help users get started and learn, and for testing applications. The driver may be downloaded and tested for free from the Ignition Module Marketplace. The module also includes a built-in manual that covers the capabilities and functionality of the module in detail.

This blog post describes some of the motivation behind the development.

I would love to see an Inductive Automation case study illustrating how an organization in the semiconductor leverages Ignition.

Module Marketplace Industry Feedback

Inductive Automation released the Module Marketplace at the ARC Forum in 2013 and in this webinar.

Gary Mintchell called it, "a platform and business model worth watching" and described the company as "developing something disruptive that is going to revolutionize the industry"

Dave Greenfield of Automation world explains the significance as "Consumer technology driving Automation" with a comparison to the iTunes store.

ARCWeb news from Craig Resnick describing "The ability to share modules with other Ignition users opens up application opportunities and removes proprietary boundaries".

Dale Peterson of Digital Bond picked it up here

I'm curious to see how developers and consumers embrace this technology. Will we see new communities form and when will other SCADA vendors follow suit?

Gary Mintchell with Inductive Automation on technology

This postcast is almost 2 years old now, but still relevant. Gary Mintchell discusses: Linux, Mobile, SQL Databases, MES and OEE, and other topics with Inductive Automation. Very insightful.

PODCAST |  

Gary Mintchell, Editor in Chief, discusses database technology, data connectivity, what today's engineers need to know, OPC UA's future and more software topics with engineers at Inductive Automation (www.inductiveautomation.com).  Approximate time: 39:22.

http://www.automationworld.com/information-management/automation-minutes-31611

Laughable security (in SCADA)

I've been preaching about the lacking state of computer security in SCADA for years; that networks will not stay isolated, proprietary, and protected.

This post really sums it up. Here's another really scary one.

The solution, at a minimum, is to apply standard IT practices to your industrial processes. Better yet, consider your architecture holistically with "Defense in Depth" in mind. In any case, have all the experts represented, which includes your IT department.

We should be able to fix this without it getting to the point of regulation.

How to Use Databases to Boost Your Production IQ

Interesting instructional video. The first guest speaker shows how to construct a query that uses functions to list the alarms that have gone off most often within a time range. Travis from Inductive Automation steps through installing a MySQL database, log PLC data, then graph it, display it in a pie chart or table in Ignition. The last presenter, Chirayu, demonstrates data collection in Rockwell Software's FactoryTalk Studio in an Energy Consumption example.

SCADA Security in the Cloud - common sense design considerations

Inductive Automation just just released this white paper on Cloud Computing and SCADA and hosted an informative webinar on the topic. I wanted to provide my perspective on a few good design practices, when looking through a security prism.

How to plan for the cloud with security in mind
- Consider the tenant of "local control". This may or may not translate to "isolated network". It's better to not even think about implementation at this point. The idea is, "from where do I want to allow my operators to control equipment"?

-Decide what it is that you need to access from anywhere. This is a business decision, not a tech decision.  

- In terms of your security planning, cloud hosting is basically the same as making your application accessible via the Internet. Do not confuse this with "public". Both approaches can provide worldwide access. Both can be secured with SSL/TLS and/or even require VPN connectivity. Consider the implications of a "worst case" compromise in the back of your mind.

- "The Cloud" is probably more robust than your server. The gold star for Availability more likely goes to the cloud. Consider that you're not taxing your bandwidth or storage resources and can pay more money to scale on an as needed basis. The whitepaper refers to this as leveraging or outsourcing IT costs. This is a good thing.

- Consider separating your Internet or Cloud services from the rest of your network, including control, business, public web, etc. I am not a fan of your cloud server being able to reach out to your PLC. Why would you do that except in the case of deliberate cloud based control, which is generally not recommended? It's just as easy to have your internal control system write data out to your online (Internet or Cloud) database. You might even log data to a local database then replicate that online. Control the flow of your communication!


Example
The point is, think about your security architecture requirements from a simple perspective. Don't worry about how you will implement it. For example, your thought process might go like this: I want to be able to change setpoints from anywhere within the control network of my 3 sites, I even need a secure way of doing this remotely. If the network goes down, local operators need to be able to control the system without missing a beat. QA needs to examine our realtime process in detail and management wants to be able to access production reports from anywhere. As you come up with this you'll inevitably realize that QA and management systems don't need a path to the PLC - that's the Principal of Least Privilege (POPL) in action! Keep up on defining requirements! Availability really matters at the operator terminals, but not for control between sites - so we don't need redundancy there. Maybe QA has regulatory requirements requiring strict Integrity (prevention of unauthorized data modification). Do we care if the management reports get out? Maybe a lot, maybe not at all. If so, we'll want to protect the confidentiality of the system. Notice that I'm focusing on what and not how?

Get the idea? Now we have our business requirements, driven by operators, management, and our process. Notice that we didn't even consider technical limitations or implementation details? Now it's time to come up with a tech plan, get with IT and let them know what are objectives are. 

Your solution may be a complex multi-part system with firewalls, intrusion detection systems (IDS), VPNs, local systems and online ones. It could also be really simple.

Closing Advice

1. First consider your business/operational requirements in the absence of naysayers who are thinking about implementation/security details.
2.  Keep security in mind as part of your requirements. This can be done without extreme paranoia.
3. Involve IT early on the design phase. Once you provide the what they should be able to help you out with the how.
4. Don't forget about upgrades/updates, backups, and other ongoing support requirements.
5. Follow the good practices that you already know. Use strong passwords, update your systems, etc. 
6. Read this whitepaper.

On water hacking

There is a lot of ongoing publicity revolving around the recent attacks on water district SCADA systems. These cases were trivial to carry out based on gaping, easily solvable vulnerabilities. Industrial controls folk need to take it upon themselves as professionals practicing due care to step it up. It isn't difficult to adopt good security practices. This isn't much different from the transition business users had to make in going from single user systems (think DOS and Windows 3.x) to networked systems. In this case, the technology exists.

It was interesting to note that an August presentation at the Black Hat conference mentioned water attacks. Coincidence? Not likely. If you work in Industrial Automation you likely know much more vulnerable and potentially dangerous targets. Let's fix this before a tragic event and regulation requires it. 

News: CNET, 
Forum discussion, and at IA, and PLCTalk.
Blog posts: Tofino II, SCADAHacker

Disclaimer - this post is not affiliated with the Water Hackathon event. The author thought it was funny.

Ignition awarded prestigious Oracle Duke's Choice Award

Inductive Automation was awarded as one of 10 annual Duke's Choice Awards from Oracle for Ignition as an innovative use of Java technology. Congratulations!!! Press release announcement.

Ignition 7.3 Beta available

Ignition 7.3 represents a major version increase for the Vision module and may significant changes for the others. If you've never beta tested Inductive Automation software you'll have to request access to the beta forum here.  I must say, it's fun to play with the new vector drawing capabilities.

• Drawing tools added for vector graphics.
• Zooming in the Designer.
• Better grouping support for components and shapes.
• New Symbol Factory module.
• More efficient serialization format for windows.
• Better color-choosing UI.
• Internationalization in Gateway/Designer.
• New compression algorithm for analog SQLHistorian tags.
• New ability for SQLHistorian to create preprocessed history tables for better query performance over long time spans.
• New query cache in the client to avoid unnecessary repeated querying of the same time span.
• Data density histogram on the Easy Chart for SQLHistorian pens.
• Improved memory usage for SQLTags in the Gateway.
• Automatic SQLTag creation when dragging and dropping OPC items.
• Improved performance and scan class settings for SQLTags (one-shot, triggered on-change, subscribed vs polled).
• Improved memory usage for ControlLogix driver.
• Improved performance and stability for all drivers.
• Improved installer allows choosing individual modules on install and upgrade.
• New graphical and command-line installer for Linux.
• Ignition installation directory structure changed.

Ignition security advisory

An independent researcher from Reverse Mode disclosed a vulnerability in Ignition from Inductive Automation. Fixes are implemented in version 7.2.8.178 and above. This finding and fix was published by the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT). The fix - install the latest version of Ignition.

It's a good idea for software companies to be transparent about releasing and fixing their vulnerabilities. Remember to keep up to date with the latest version of your software! 

Interesting article - #1 ICS and SCADA Security Myth: Protection by Air Gap

Eric Byers of Tofino Security (they do SCADA security research and have a security appliance) discusses the concept of relying on air-gapping control networks for security. In a nutshell, a modern control network is unlikely to remain physically separated. Even if it is, malware can be introduced via removable media and the system is hard to keep updated and vulnerable to local (insider) attacks.

A system should be designed with security in mind and kept up to date. 

http://www.tofinosecurity.com/blog/1-ics-and-scada-security-myth-protection-air-gap

Interesting article - The Ultimate HMI

This Control Global article, The Ultimate HMI, provides interesting tips about HMI screen creation. I especially like the comments and examples of the best visual status indications containing: text, shape, and color.

Ignition intro - Rethinking SCADA for modern manufacturing

An informative 3 minute video of the Ignition concept from Inductive Automation, rethinking SCADA for modern manufacturing.

Industrial Security - a reminder from Stuxnet

I've refrained from blogging about Stuxnet for the simple reason that I'm no expert on it. You can learn more than I know about Stuxnet from Wikipedia. However, it should reinforce the need for basic security good practices. I hear too many integrators and industrial professionals write off a system as not needing any security applied because the system is on an isolated network. Consider the vector Stuxnet used to attack isolated networks - USB drives to infect the system then peer to peer Windows RPC calls. Even if you're not the target of a large attack, you need to protect yourself from the much more common "insider threat", a disgruntled or malicious employee. Let's consider some easy steps that can make a huge difference in protecting your SCADA system and industrial network. 
1. First, get the best "bang for your buck" - take care of your low hanging fruit. Embarrasingly, in industrial systems this means: get rid of shared accounts and no passwords, don't connect the control network to the Internet or harden your business necessary connection point, ensure that you have a working backup, check your backdoors (old modems, etc).

2. The strength of your system lies in a Defense in Depth approach, meaning taking advantage of strengths of overlapping security mechanisms. While adhering to point #1, knock out the really easy and obvious ones. There's no reason to get fancy until you have your basics covered.

3. Architect your system with security in mind. For example, if you decide to start with Windows XP SP 1, you're just asking for trouble. I get it that patching often has negative side effects with industrial/SCADA software, but you need to (minimally) begin with a baseline that isn't hugely vulnerable. Design around industry standard technologies and protocols such as OPC-UA. Don't try to roll your own security solutions!

4. Develop a security policy and train your users. It's important that management understands and accepts the level of risk that the system takes on. It's important that users know what's allowed and that consequences are serious. Require that they sign an Acceptable Use Policy (AUP). A successful security program is as much about users, policy, and procedure than equipment, applications, and configuration.

5. Establish individual accountability (auditing). Ensure that users have individual accounts and that their activity on the system is logged and periodically reviewed. I know this often doesn't happen in manufacturing, but it should.

You have far too much value in your HMI, SCADA, industrial system. Ask yourself, how much does downtime cost? How can you afford to not secure your control system?

Going mobile with industrial computing

Mobile is hot in manufacturing! The web sites of the major SCADA vendors, industry blogs, and consumer feedback indicates a strong interest in "mobile", be it tablet, PDA or smartphone. Who wouldn't want the capability to see, if not control your process from the palm of your hand? Why deal with text messages, phone calls, or email alerting when you can see, acknowledge, and handle alarms from your mobile device in real time? What about tying this to your MES/ERP system to view production efficiencies?

Sure there are red flags - you will probably have an increased requirement for security including physical access, user accountability (auditing), and all the typical considerations. For most organizations the benefits will most likely outweigh the risk and cost. Security is a mature enough field - if the credit card companies, banks, financial institutions, and even the military can be successful with it, so can you. Apply defense in depth, implement/follow your company policy, adhere to strong practices, hire experts if you need it - nothing new here.

From my perspective, industrial computing has been following the path of commercial computing. Those same basic technologies that have become widespread in your office will usually show up on the plant floor in some form. You can trace industrial controls from relays to PLCs as opposed to vacuum tubes then transistors, but the technologies have been converging and are basically one in the same.

Perhaps a driving force of computing paradigms is the relative shift in local processing power versus the capability of the "network". How thick versus thin is the ideal environment - in other words, do we want to perform the computing locally or remotely? We've seen this shift between centralized and de-centralized SCADA systems - the balance between central administration versus local setup, scalability and performance, and dependence on the network. The models roughly share the considerations we've seen in the migration between: Mainframe, Micro, Mini, and PC.

So where are we going next? Cloud computing or even thinner clients? To me, mobile computing represents an advance in both categories. That's not to say that smart phones won't get more computationally capable - they will continue to shrink and get faster and faster. However, with the increasing reliability of infrastructure, "the network", it makes more sense to centralize your control applications. Why? In one word, ubiquity. For the same reason the web has standardized on HTML. Users will demand access from anywhere they deem appropriate, regardless of the mobile platform. Will they create a separate app for each: iPods/iPads, Blackberry, Droid, Win 7 phones, Palm PDAs, embedded MS OSs, Linux devices, etc, etc? Each of those platforms has a large enough user base to support its own "apps" - Solitare, or a web browser, which turns out to be the common denominator.

I don't think organizations will put up with only supporting one mobile platform. This is somewhat akin to requiring you to buy a Bell phone in order to use the phone network. Nor is it acceptable for a complex installation or configuration. Successful SCADA vendors will need to support feature rich mobile applications that run on most, if not all, mobile devices out of the box. As our networks get more robust, secure, and cheap, we'll be rewarded with new levels of capability that make that transition from novelty to necessity - just like PCs once did. In the end, competitive organizations from all industries will benefit from these incredible mobile technologies.

Computing Without Boundaries

Steve Hechtman, the President of Inductive Automation, has been busy updating his blog, Computing Without Boundaries. It's a fantastic, albeit biased, prospective on modern technologies, methodologies, and considerations for modern Industrial Software applications. Taken with a vendor neutral approach, his points focus on today's issues and technology on the forefront. He isn't afraid to lay it down with points of frustration, digging into his 26+ years as an Industrial Integrator, which can be quite entertaining as you relate it to common situations. He sometimes relates the approach that Inductive Automation uses, justifying exactly why it makes sense in each case - to the Integrator and end user. Definitely worth a read. In fact, leave a comment or two.

Retro Encabulator interview

The Retroincabulator video makes me laugh every time I see it. Tim managed to get an interview with actor Mike Kraft about it. Sweet!

Review
Interview

VCP at last, and virtualization in Industry

I had the opportunity to attend VMWare's Fast Track course last December. It was a good class. The info came fast and furious! After nearly allowing my test voucher to expire, I buckled down and hit the books.On Wednesday I passed the test, so now I'm officially a VMWare Certified Professional!

Virtualization has been an interest of mine for awhile. I live by VMWare Workstation on my desktop and await the day when I get a laptop that's powerful enough to support my too-many VMs at once habit. For the Industrial Integrator/programmer/professional, Virtualization makes too much sense! Check it out if you haven't already! It readily solves problems that Industrial Software vendors have created (you need this version of Windows without a patch, can't run with competing software, older programming tools for some customers, etc, etc). Using multiple Virtual Machines, you can have exactly the right tool for whatever task at hand - without ever needing to make any changes. You get additional benefits of features like "snapshots" and "cloning". The only real issue is that none of the vendors have decided to support PCMCIA/PC Cards, so you'll have to use serial or USB connectors. This should matter less and less as time goes on.

On the server side (which is what the VCP is all about), VMWare may or may not make sense for your Industrial applications. Don't get me wrong - ESX and ESXi are awesome packages, but if you don't have a competent IT department - you're adding another layer of complexity for what might benefit you. For those larger companies that provide a lot of server based services - you're probably already using a virtualized environment. It just doesn't make sense not to.

Web based SCADA - a perspective from the Netherlands

Web based SCADA article by Eduard van Loenen of Yokogawa’s Global SCADA Center in The Netherlands. Kudos to Dan for pointing it out. He brings many great points to the table of the advantages to the trend of web based SCADA systems. 

From a security perspective, I disagree that IT technologies are "more susceptable to cyber attacks" than "rigid proprietary software". The cyber attacks will come since systems are being networked and remotely accessbile, which is a productivity enhancing user requirement in modern manufacturing. Proprietary software tends to be less secure than modern IT technologies, especially as the Internet makes exploits more widely available.

Good article, though!

Databases – The Perfect Complement to PLCs, by Steve Hechtman

Interesting insight on the role of SQL and databases with PLCs and industrial automation.

Hosted at Automation.com,  Reliable Plant, FoodEngineering

Ignition, post release activity and OPC vendors

I was impressed with the Ignition by Inductive Automation product release webinar. The Inductive Automation team did a first class job introducing the company, Ignition product platform and architecture, and the motivation behind the shift to OPC-UA. In a nutshell, their vision of the future of industrial software is: standards based, IT friendly, and platform independent. Get DCOM out of here! We want a top notch, secure SCADA system that runs equally well on Linux, a mac, or any flavor/service pack of the Windows rainbow. The demonstration included a full server installation, which took about 2 minutes, and highlighted the drag and drop ease of the new SQLTags History feature, which is deceptively simple, yet powerful.

We've been getting a huge number of hits to the new website by a number of companies across the world. We've also been getting a lot of interest in when more drivers will be available, like Siemens, Modbus, and Automation Direct in addition to the existing Allen Bradley suite.  I suppose this isn't surprising considering that we're the only cross-platform OPC-UA server available. While it shouldn't be too long before we get those drivers developed, this brings me to my next point...

Where are the other OPC-UA servers? We're really looking forward to a more vibrant marketplace for OPC-UA products. Kepware says they'll be releasing their OPC-UA server soon - we're looking forward to being able to test it with Ignition. This really excites me because it brings their extensive PLC product line to the table via OPC-UA. Unfortunately, I don't think it will be cross platform.

So, how about it? OPC vendors: the gauntlet has been thrown. The industry wants OPC-UA products! And if they're cross platform, all the better. When will we start seeing them?

My thoughts on securing your plant systems

Security, not your first concern as an industrial integrator or plant manager. Availability ranks numero uno. Got it! Your company loses $30k/hr during plant downtime. Got it! Plant workers are not exactly NSA hacker material. Got it! So why is security important? Look at those reasons again and above all, safety. Consider that your legal liability in court costs more than downtime. Do you want a disgruntled employee to shut down the factory on your watch? It is incumbent upon you as a professional to demonstrate due care. We're not worrying about international hacker rings or bored young college geniuses. Accidents occur and insider attacks are possible.

It's 2010 - many systems end up touching the Internet or outside world somehow, whether you like it or not. Common practices in the 90s, like allowing users to share passwords are unacceptable - that audit trail is a must. Having PLCs on the same network as office computers - professionally irresponsible. Your company uses 25 year old hardware that was never designed for security - it's up to you to isolate that network. Provide access with a "hardened" dual-homed (2 network cards) computer that is patched and protected by a firewall. Utilize VPNs, DMZs, VLANs, SSL, IDSs, and anything else in your IT department's arsenal. They know how to secure a network - it's their job. Gone are the days of operating behind their back, not letting them touch anything for fear that updates will break your system. Pick a vendor that IT will support. Practice Defense in Depth. Let them help you. It's 2010. Embrace positive change!

Introducing Ignition by Inductive Automation - Web-based, cross-platform SCADA Webinar

From an official Inductive Automation email....


Say goodbye to rationing clients, patchwork control systems and project development constraints caused by oppressive licensing systems.

Ignition by Inductive Automation is being released January 27th. Ignition is a major next step in the evolution of FactorySQL and FactoryPMI.

Join us for this new release webinar to see how the Ignition platform opens up possibilities in the industrial automation industry. The webinar will give an overview and demonstration of Ignition's features and its capabilities, such as:

• Web-based clients that enable scalability
• Unlimited clients and tags, licensed by the server
• A flexible database-centered architecture
• OPC-UA support
• 100% cross platform: Windows and Linux
• Simplified software setup and ...

Also accompanying the product launch is Inductive Automation's new website design at http://www.inductiveautomation.com/, which will launch January 27th with details about Ignition.

Register here to attend the new release webinar.

Title: Introducing Ignition by Inductive Automation - Web-based, cross-platform SCADA
Date: Thursday, January 28, 2010
Time: 9:00 AM - 10:00 AM PST

After registering you will receive a confirmation email containing information about joining the Webinar.

System Requirements
PC-based attendees
Required: Windows® 2000, XP Home, XP Pro, 2003 Server, Vista
Macintosh®-based attendeesRequired: Mac OS® X 10.4 (Tiger®) or newer

Space is limited.
Reserve your Webinar seat now at: https://www2.gotomeeting.com/register/273245266

Ignition preview and a visit to Inductive Automation

Over the holidays I had the opportunity to visit my good friends, Colby and Carl, who are lead developers at Inductive Automation. They gave me an in depth technical "sneak peak" of what is becoming their next product line, "Ignition".

The plan is a single product platform that replaces the separate functionality of FactorySQL and FactoryPMI. They really went all out with levering technologies! The product is platform independent (Windows 32 or 64 bit, Linux, Mac, etc), fully web based (no client installation), and supports separate "modules" that all have access to the same powerful core technologies. Everything runs from a cluster aware web server similar to the current FactoryPMI Gateway but they had new behind the scene tricks (Non-Blocking IO for multithreading, advanced serialization schemes for saving/storing projects, and a whole slew of security upgrades consistent with the OPC-UA specification).

They showed me an OPC-UA application that securely connected to different PLCs (many AB, Siemens, and AD) via different PCs without an OPC tunneller. All this worked equally well on Windows and Linux systems. Once the data is accessible from the PLC, you get the full visualization spectrum that FactoryPMI already offers (graphics, multimedia, historical trends, reporting, etc).

The coolest part is that development time should be cut down substantially, for both Inductive Automation proper and 3rd parties/end users as the API is released, since the platform performs so much native functionality (network, database, project launching/hosting/editing, clustering, storage, alerting, auditing, authentication, etc, etc.). The Ignition platform should become to controls software what Metasploit is to ethical hackers/penetration testers.

OPC UA info

I just ran into this article dated May of last year. Simone Massaro of Iconics describes the direction that they went with OPC UA development. It's a bit technical, but a good read.

http://www.plantengineering.com/article/talkback/183110-What_is_OPC_UA_and_how_does_it_affect_your_world_.php

Is anyone buying this? Really...?

I stumbled across this article that, quite honestly, at first pissed me off. After a little reflection, I can only laugh. It reminds me of someone trying too hard to sell those $150, short gold plated digital Monster audio cables - Oxygen free or whatever. (If the engineer inside you doesn't laugh, then cry for the sake of the suckers, read on).

Maybe I read in to far, but I see snakeskin oil vendors grasping for air! The piece is an obvious response to Steve Hechtman's very different article on the same topic (hosted on Control Engineering) - he should be flattered. You see, the big vendors, GE Faunc in this case, but the exact same applies to Wonderware and Rockwell have been long committed to the concept of Historians, a glorified and expensive datalogger that includes, and is only meant to work with, a custom version of Microsoft SQL Server (of all product choices...). The problem is that now much cheaper products from companies like Inductive Automation and Software Toolbox can do a better job using any RDBMS (database) system. Being vendor-neutral, inexpensive add-on packages also do much better for things like trending, reporting, and data analysis. The biggest mistake of the current generation of Historian is that they tried to implement and include everything themselves - like making a giant Swiss Army knife with a spork, usb memory stick, and a wine glass. Now they're caught with their pants down, desperately scrambling to recovery their enourmous sunk costs (my favorite business term).

• You're data is special and requires "plantwide historian" treatment...their example query “What was today’s hourly unit production average compared to where it was a year ago or two years ago?”. I won't even comment...

• Your database needs to speak specialized industrial protocols (OPC) - There's separation of function by design and for a reason. Besides - this doesn't even make sense.
• Faster speeds and higher data compression - no way! The historian is wasting CPU cycles in both directions, which obfuscates your data (can no longer use external applications), to do something better achieved by a RDBMS system that supports it.
• Robust redundancy for high availability - is this a joke? Maybe Amazon.com should migrate their server farms over to GE-flavor SQL Server.
• Enhanced data security - another losing battle for the historian. The white paper mentions SQL injection attacks - all platforms in question can use stored procedures, and are all subject to this sort of attack. When it comes to up to date patching, arguably the most common vulnerability, SCADA vendors have the absolute worst track record! IT keeps their servers patched as a matter of practice - they're typically afraid to touch the SCADA machines. Ultimately, the "do everything" approach provides many attack vectors.
  

I can't blame them for playing their hand. I just wonder - will anyone read this white paper and take it at face value?