What you learned in school - applied to Sierra Nevada Brewery

I remember some discussion in my recent IT classes about the "network effect", an idea that academia uses to describe a good or service whose value increases as more people/nodes have access to it. For example, a technologically more advanced social networking site would likely run into a brick wall against Facebook due to their existing customer base. 

It took me a bit to put two and two together to realize that this same "network effect" was the driver in the Sierra Nevada Brewery case study video that I recently commented on. In this case, the effect applies to access to accurate, timely, information within the organization. For example, the QA department benefits from the fact that Fermentation and Filtration use the same system, even for their own purposes.  

This case study highlighted the value of obtaining IT support, whereas the norm for industrial users tends to be a struggle. The other point was enterprise wide workflow integration, where their initial pilot system was deliberately created in two areas that couldn't slow production - data acquisition and reporting. 

It's cool to see when an organization can access its data centrally and apply that to the business flow across departments. This is a microcosm of what I understand is a significant aspect of what the Digital/Internet Revolution is supposed to bring.  

Ignition paving the way with alarming - 7.6 public Beta released

The public Ignition 7.6 beta was just released. It adds a bit of complexity and brings a whole lot of capability. The main feature set is an alarming revamp that I believe pushes the envelope for automated SCADA notification. I'm not sure how many of the individual features are "firsts", but I suspect a few. For example, SMS text messaging and email both support 1 or 2 way communication, meaning you can acknowledge alarms by replying to a text message or email. Voice enunciation is supported via the SIP protocol, giving you the option of using a commercial Skype service, a cheap "asterisk" appliance box plugged into a phone line, or your existing commercial VoIP setup. The old "Intel/Dialogix/Win 911" solution I used with RSView years ago for a small hydroelectric plant did this enunciation in a cool way, but was awful for numerous reasons. The Ignition system runs everything through the "store and forward" system, which caches all historical information if connection to the SQL database is lost. It also stores all the alarm logs, system logs, and historical data in a consistent format so you can leverage pre-canned widget viewer objects. There are lots of different deadband/delay/grouping options to manage flapping values on the border or simultaneous sets of alarms. Flexible schedules and rosters are available. Most interesting, is the notion of "Alarm Pipelines", which are graphical representations of complex alarm logic. You drag and drop a flowchart to create a potentially complex system of: notifications, expressions, set properties, delays, and jumps to other alarm pipelines.

Some of the Ignition beta documentation:

Ignition 7.6 - Alarming - Beta (In progress)

Ignition 7.6 introduces a completely redesigned, feature rich alarming and notification system aimed at tackling all of your complex SCADA alarming needs.  These new features are the culmination of many years of customer feedback, as well as in depth research into SCADA alarming best practices and should put Ignition well in the lead industry-wide for alarming capabilities.

Overview

System-wide changes, from the Ignition platform to the introduction of new modules, have taken place in order to best incorporate the new alarming features in a seamless fashion into the Ignition software ecosystem.  Each of these changes are addressed in detail throughout this document, but to begin with let us take quick, higher level look at the new architecture.  The hope is to gain a little familiarity with new terminology and a loose understanding of how all of the pieces are tied together before diving into detailed explanations of how everything is configured.

Pieces of the Whole

As you can see from the above architecture diagram the new alarming system can be divided up into three major areas (Ignition Platform, Vision Module, Alarm Notification Module) with each area being home to several pieces of the new alarming system.  Each of these pieces are interconnected and provide different pieces of functionality to the alarming system.  

Ignition Platform

1. Tag Configuration - You’ll notice right away when editing a SQLTag’s alarm configuration that this has changed dramatically in 7.6. Tags can contain multiple alarms. Each alarm’s properties can be configured to be dynamic via binding them to tags or expressions. Associated data can be added to an alarm, allowing for custom organizational structure or adding process data to be stored along with the alarm event. All previous alarm configuration will be automatically converted when upgrading to Ignition 7.6.

1. User Sources - This is where you configure users (both for authentication purposes and notification purposes) and roles in Ignition. If you’re a user of Ignition 7.5 or earlier, you’ll notice that this is an evolution of the Authentication Profile system. User sources are responsible for providing Ignition users along with their associated username, password, and contact information needed for notification.
2. Schedules - Schedules define day and time availabilities that you assign to users.  They are used for determining when a user is available to be notified of active alarms.
3. Alarm Journal - The new Alarm Journal system replaces the old alert log. This journal logs all alarm activity to an external SQL database. The table structure is managed by Ignition. Querying is mostly done through new scripting functions. Querying can also be done directly against the database if need be.

Alarm Notification Module

1. Pipelines - Pipelines are a new construct that define the logic between an alarm going active/clear and people being notified. They logic graphs and are configured in the Designer via a drag-and-drop UI. Pipelines are where you can add logic like delays, message consolidation, escalation, etc. Note that pipelines are the “heart” of the notification system - you need this feature to do any alarm notification.
2. Email Notification - Email notification is included in the Alarm Notification Module. Email now has the capability to support acknowledgement as well.
3. Upgrades/Licensing - Note that the alarm notification is a new module, but since it covers functionality that was previously in the platform, users who are upgrading from pre-7.6 who are covered by our upgrade protection will get this module comped as an “upgrade”. This module will retail for $1,995.

Vision Module

1. Alarm Status Table - The vision module has a newly renovated alarm status table. It is much more customizable than the old table: row styles are dynamic and configurable via a customizer, columns are configurable as well. It has an ad-hoc chart feature for tags using the SQLTag Historian. This is also where the shelving feature is accessed.
2. Alarm Journal Table - There is now a built-in table for viewing alarm history, now called the alarm journal. You’ll find this component on the Tables tab of the component palette. You’ll probably want to bind the start/end dates to something appropriate, like a date range component.
3. Upgrades / Licensing - The major version number of the Vision module has not changed, so this does not constitute an upgrade.

SMS Notification Module

1. SMS Notification -  This module adds the ability to notify users via SMS. This is done using an Airlink Raven XE from Sierra Wireless. Communication between Ignition and this cell modem are over ethernet.
2. Upgrades / Licensing - This is a brand new module, so upgrades do not apply. This module will retail for $595 and requires the Alarm Notification Module.

Voice Notification Module

1. Voice Notification - This module adds the ability to notify users via a phone call. It uses a TTS (text-to-speech) engine to turn alarm messages into audio, and uses VOIP (specifically SIP) to make phone calls. If you do not have a SIP gateway on premise, you may use a commercial Skype account (requires internet) or an Atcom IP01 FXO gateway which will convert SIP to a plain old telephone line. (http://www.atcom.cn/IP01.html)
2. Upgrades / Licensing -  This is a brand new module, so upgrades do not apply. This module will retail for $695 with one voice, additional voices $99, and requires the alarm notification module.

Voice Alarming

The new Phone Notification Module adds the ability to deliver alarm notifications to users via telephone, using any SIP compatible phone system. Messages are constructed in text, and are delivered through a high quality text-to-speech engine. The engine supports multiple voices and languages.

Core Features

1. Deliver voice calls through any SIP compatible phone system. No dedicated hardware required.
2. Messages generated by high-quality text to speech, and not a canned set of prerecorded files.
3. Supports multiple languages concurrently, based on user preference.
4. Allows users to acknowledge events.
5. Supports requiring a personal identification number for additional security.
6. Ties into the audit log to audit call events, successful message delivery, and user acknowledgements.
7. Supports throttled message delivery.

Using Skype

If you don’t have an existing VOIP system in place, using a hosted service is the quickest way to get going with the voice notification module. Skype offers SIP based service through their Skype Connect product. To get started, you must have a business account, which provides you access to the Skype Manager. From there, you can create a new Skype Connect channel. For more information, visit http://manager.skype.com

Once you have created a Skype Connect channel, configuration in the gateway is similar to that of any other PBX system. The host will be “sip.skype.com”, and the user name and password will be those provided by skype during the registration process.

Schedules and On-Call Rosters

Schedules and On-Call Rosters are new to Ignition 7.6 and allow you greater control over managing who should be notified and when. These two elements work together when it comes to sending out alarm notifications to users.  

Schedules

Schedules define when a user is available to be notified, and rosters create a grouping of users that should be notified when certain alarms pass through different notification blocks in your alarm pipelines (see the alarm pipelines section below for an in-depth explanation).  Each notification block in an alarm pipeline has an On-Call roster property that associates the block with a roster.  When an alarm passes through that block the alarming system begins to work its way through the members (users) of the specified roster and notifications are sent to the users whose schedules list them as being available.

Alarm Pipelines

Alarm pipelines are something new to Ignition 7.6.  No longer is the logic for when, how, and to whom notifications are sent contained in the gateway.  As detailed above, notification profiles themselves have been stripped entirely of any functionality that doesn't have to deal directly with the delivery and receipt of notifications and acknowledgement.  All notification logic is now designed/specified and edited in the designer by creating Alarm Pipelines.  There is a dedicated workspace in the designer for pipeline construction that allows you to create and manage any number of unique pipelines.

Pipelines aim to give the user an easy to use, drag and drop mechanism for creating complex notification scenarios that allow for the incorporation of key features that were previously unavailable. Alarm escalation, notification throttling, dynamic/custom properties for alarm events, all of these features are now made available through Alarm Pipelines enabling you to customize pipelines to fit all of your alarming and notification needs. Understanding the flow of alarm events through a pipeline and how exactly pipelines work is key to being able to fully exploit the potential of Alarm Pipelines.

Alarms are configured on Individual SQLTags and each alarm is tied to an Alarm Pipeline. When the condition for that alarm is met a new alarm event is generated and sent to the entry point of it’s associated pipeline.  The alarm event then moves through the different logical elements of the pipeline until it finally reaches a pipeline endpoint upon which it then exits the pipeline.  There can be multiple alarm events in a pipeline at any given point in time and you can view the current status of the pipeline from the Alarm Pipelines status screen in the Ignition Gateway.  Some pipeline objects let an alarm event exit the pipeline it is currently in and jump to another pipeline entrypoint allow alarm events to be redirected when certain conditions are met. Once the alarm event reaches an endpoint of a pipeline that does not redirect to another pipeline it drops out then that alarm event’s existence comes to an end. 

Pipeline Blocks

There are six types of pipeline blocks that are used to construct your pipelines and each are used to accomplish specific tasks: Switch, Jump, Expression, Delay, Notification, Set Property, and Splitter.

Alarm Properties

Main -

1. Name - A unique name for the alarm
2. Enabled - Enables or disables the alarm
3. Priority - Choose from five priorities, from Diagnostic to Critical, for the alarm
4. Active Pipeline - Sets the pipeline to use when the alarm goes active
5. Clear Pipeline - Sets the pipeline to use when the alarm clears
6. Timestamp Source - Whether the timestamp comes from the system or from the value
7. Display Path - Optional path that will be used for display.  Takes a string separated by “/”
8. Ack Mode - Auto: Alarm is automatically acknowledged when it clears; Manual: Alarm must be acknowledged by the user; Unused: The alarm will always be marked as acknowledged
9. Notes - Optional notes regarding the alarm
10. Ack Notes Required - When set to true operators will be required to enter some explanation when the alarm is acknowledged
11. Shelving Allowed - Whether or not the alarm can be shelved (snoozed)

Alarm Mode Settings

There are ten different alarm modes to choose from.  The mode you select will determine any additional Alarm Mode Settings properties to be displayed.  Below are the different modes and their relative additional properties:

1. Equal

1. Setpoint - In this mode when the tag value is equal to this setpoint value then the alarm will be considered active.

2. Not Equal

1. Setpoint - In this mode when the tag value is not equal to this setpoint value then the alarm will be considered active.

3. Above Setpoint

1. Setpoint - In this mode when the tag value is above this setpoint value then the alarm will be considered active
2. Inclusive - When true then alarm will be considered active when the tag value is equal to or greater the setpoint value

4. Below Setpoint

1. Setpoint - In this mode when the tag value is below this setpoint value then the alarm will be considered active
2. Inclusive - When true then alarm will be considered active when the tag value is equal to or less than the setpoint value

5. Between Setpoints

1. Low Setpoint/High Setpoint - These setpoints create a range that the tag value must fall within in order for the alarm to be considered active.
2. Low Inclusive/High Inclusive - When true then alarm will be considered active when the tag value is equal to or less than/greater than the respective setpoint value
3. Any Change - When true any change in tag value that falls within the range defined by the high and low setpoints will generate an alarm event that isimmediately cleared. Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

6. Outside Setpoints

1. This mode operates similar to the Between Setpoints alarm mode.  You configure a range using the low and high setpoints and when the tag value falls outside of this range then the alarm is considered active
2. Any Change - When true any change in tag value that falls within the range defined by the high and low setpoints will generate an alarm event that isimmediately cleared. Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

7. Out of Range

1. Generates an alarm event whenever the tag value falls outside of the Eng Limits defined in the tag’s numeric section

8. Bad Quality

1. Alarm events will be generated whenever the tag reports having a bad quality

9. Any Change

1. Alarm events are generated whenever the tag value changes.  These events are immediately cleared and if the Ack Mode is set to auto they will also be auto acknowledged.  Be aware that by default dropout conditions on pipelines include when an alarm event is acknowledged and or cleared.  This means that if left at the default settings you will not receive notifications for alarm events generated by the Any Change setting.  This behavior of alarms generated by any change events may change before the final release of 7.6.  Make sure to check documentation for any changes when 7.6 is officially released.

10. On Condition

1. Is Active - This is a bindable property allowing you to specify an expression that will evaluate to true or false and determine whether or not the alarm should be active.  You also have the ability to bind this property to another driving tag.

Deadbands and Time Delays

1. Deadband - A numeric deadband for the alarm condition.  This deadband is separate from and evaluated after the tag’s deadband.  An alarm event will be generated as soon as the tag value crosses the active threshold but will only clear after the tag value goes outside the active range by more than the deadband.
2. Active Delay (rising edge time deadband)- The time in seconds before the alarm will be considered truly active after the active condition evaluates to true.  If a tag value crosses the active threshold and then clears before the delay period elapses then an alarm event will not be generated.
3. Clear Delay (falling edge time deadband) - The amount of time in seconds before the alarm will be considered truly clear after the clear condition evaluates to true.  If a tag value goes into the cleared state but then becomes active again before the clear delay period elapses then the alarm will stay active.

Sierra Nevada Brewery MES, ERP Ignition Case Study - Awesome!

This 5 minute case study video illustrates how Sierra Nevada Brewing Company ended up using Ignition as their MES and ERP platform. "We use it for everything as our go-to product". They do most of their design work in house, constantly updating their "slightly unique" process.

It's interesting to see the evolution of the importance and use of the brewery's SCADA system. They started with a small pilot project to capture PLC data, basically just as a stand alone OPC data logger. Another department tried out that same server as an HMI. The video shows how the demand signal for Ignition flowed from department to department by word of mouth. "Once it proved itself we started adding". Everyone really liked it - they added their section and wanted access to the growing realtime database: the brewhouse, fermenters, the cellars, filtration, the lab, packaging, tracking trucks, Quality Assurance (QA). Even their IT department.

The Ignition system grew with fan-based requests. The more information was available the more users craved. Everyone wanted to add their devices, sensors, and part of the process.  Once users see what's possible, they want a lot more: reports, charts, graphs, realtime and historical information. Sierra Nevada beer is tracked from tank to tank, information like "yeast cell counts", "CO2 values", and "temperatures", recorded automatically, which represents the production phase that the batch of product is in. They used to walk around the facility to track their beer through whiteboards. Now all of their information is tracked in their SQL databases, accessible from any desktop computer. Pretty cool!

Ignition in many industries

Inductive Automation 48 second teaser video of the many industries that Ignition SCADA software is used.

Appliances, automotive, building material, chemical, electronics, energy, fibers & textile, food & beverage...the list goes on.

Ignition and SCADA in the semiconductor industry?

I was poking around the Ignition Module Marketplace and noticed the "SECS/GEM" driver module for their OPC-UA server. Curiosity got the best of me and I was soon reading about the Semiconductor Equipment Communications Standard and Generic Equipment Model. While I find it a bit curious that the semiconductor industry, specifically SEMI (Semiconductor Equipment and Materials International) is big enough, or has some set of semiconductor unique requirements that justify an industry proprietary protocol. I remember a story in college from an "Intel Fellow" who headed up the early Pentium IV project. He described a colleague applying chaos theory to the sequencing of these $50 Million "machines" that would treat silicon wafers for weeks. My assumption would have been that, had existing standard protocols and standards not been sufficient, that the scale of semiconductor operations justify custom code to be written. I had not considered the semiconductor industry as another SCADA application. Thinking back, I remember how I envisioned automated manufacturing before ever doing any integration work. A giant factory rapidly creates parts and assembles pieces. Everything whizzes rapidly through an automated assembly line without any user intervention. One or two workers walk the floors with hard hats and clipboards to watch the magic happen. Does this sound like your process? "Real" facilities are a bit different. People are involved in the process, often continually changing it. Equipment must be integrated to work together. Budgets and efficiency are more important than ever. Simply put, most organizations that are smaller than Walmart cannot remain competitive by hiring teams of professional programmers to support their operations. Why would the semiconductor business be any different? I realize that my lack of experience in the field led me to put it on a pedestal. That industry can benefit from the rapid development capability that Ignition provides and, apparently, an $895 SECS/GEM driver. That's an exciting thought - I love to see the extensibility of the Ignition platform in action. From their web site:


The SECS/GEM Module enables Ignition projects and third-party applications to communicate with semiconductor fab equipment. The module utilizes the HSMS-SS and SECS II standards to communicate with equipment over Ethernet. In the near future, SECS I for serial connections will be supported as well. The module achieves interoperability with third-party systems through the use of shared database tables. For use with Ignition software – a rapid application development and deployment platform – built-in mapping to tags or scripting functions can be utilized. Driver configuration is done with a web-based configuration interface which includes a built-in editor for creating custom SECS messages. Over 100 SECS message definitions from the SECS II and GEM standards are supported out of the box. Ignition and this driver are cross-platform capable because both were developed in Java. An equipment simulator is included with the module to help users get started and learn, and for testing applications. The driver may be downloaded and tested for free from the Ignition Module Marketplace. The module also includes a built-in manual that covers the capabilities and functionality of the module in detail.

This blog post describes some of the motivation behind the development.

I would love to see an Inductive Automation case study illustrating how an organization in the semiconductor leverages Ignition.

Module Marketplace Industry Feedback

Inductive Automation released the Module Marketplace at the ARC Forum in 2013 and in this webinar.

Gary Mintchell called it, "a platform and business model worth watching" and described the company as "developing something disruptive that is going to revolutionize the industry"

Dave Greenfield of Automation world explains the significance as "Consumer technology driving Automation" with a comparison to the iTunes store.

ARCWeb news from Craig Resnick describing "The ability to share modules with other Ignition users opens up application opportunities and removes proprietary boundaries".

Dale Peterson of Digital Bond picked it up here

I'm curious to see how developers and consumers embrace this technology. Will we see new communities form and when will other SCADA vendors follow suit?

Gary Mintchell with Inductive Automation on technology

This postcast is almost 2 years old now, but still relevant. Gary Mintchell discusses: Linux, Mobile, SQL Databases, MES and OEE, and other topics with Inductive Automation. Very insightful.

PODCAST |  

Gary Mintchell, Editor in Chief, discusses database technology, data connectivity, what today's engineers need to know, OPC UA's future and more software topics with engineers at Inductive Automation (www.inductiveautomation.com).  Approximate time: 39:22.

http://www.automationworld.com/information-management/automation-minutes-31611

Laughable security (in SCADA)

I've been preaching about the lacking state of computer security in SCADA for years; that networks will not stay isolated, proprietary, and protected.

This post really sums it up. Here's another really scary one.

The solution, at a minimum, is to apply standard IT practices to your industrial processes. Better yet, consider your architecture holistically with "Defense in Depth" in mind. In any case, have all the experts represented, which includes your IT department.

We should be able to fix this without it getting to the point of regulation.