Oh my - FDA 21CFR11

Read the reg!

I don't know whether to laugh or cry at Title 21 of the Food and Drug Administration's Code of Federal Regulation Part 11. It dictates requirements for Electronic Documents and Signatures. It seems to written on technical topics by non-technical people. It's well intended, but deliberately vague on implementation standards - over the top in many cases. The funny thing is that most vendors tout a "Certified 21CFR11 compliant" stamp on their hardware or software. The sad thing is that end users fail to realize that 80% or so of the requirements have to do with their: processes, training, documentation, etc.

I was getting so frustrated by a couple of users asking specifically how to implement a 21CFR11 application in FactoryPMI. They wanted point by point descriptions all the way down the list. I was first approached in 2005, but don't know of any users who have actually attempted a fully compliant project. I've had other users working in a 21CFR11 "compliant" facility that just wanted to use FactorySQL or FactoryPMI beside the application. I had to bite my tongue when one said that a hardware datalogger satisfied that requirement. Did anyone there actually read the regulation? Oh gosh! Maybe a good integrator helped them make their whole process compliant. I know for a fact that you can't throw any single piece of hardware or software and make it happen. Things like 11.300(e):

Initial and periodic testing of devices, such as tokens or cards, that bear or generate identification code or password information to ensure that they function properly and have not been altered in an unauthorized manner.

Carl made me feel better when he pointed out that 21CFR11 isn't a compliance standard for HMI/SCADA design software. Its a compliance standard for HMI/SCADA project implementations, which are two totally separate things. So, I wrote a white paper on creating a compliant 21CFR11 application with Inductive Automation software and didn't feel so bad about all the, "it is the responsibility of the designer..." answers. I might add that the "big vendors" white papers have similar responses.

I'd like to see someone actually do this - but it'd take planning on the front end and someone with experience. It would be absurd for an Integrator or end user to take this on from scratch, without help, as a first project.

Fascinating interview with Sun CEO

I was recently directed to read an interview with Sun Microsystem's CEO/COO, Jonathan Schwartz regarding his Open Source business strategy. This is particularly relevant given their new MySQL acquisition.

http://blogs.cnet.com/8301-13505_1-9757417-16.html

Jon stressed the necessity of driving adoption to generate revenue. He claims that seemingly competing products like Red Hat and Solaris complement each other because both are going for the same goal - look out M$! He makes a distinction between his "community" (users) and "market" (paying customers), insisting that one leads to the other. A casual user or developer shouldn't pay for his software, while that same individual might recommend purchasing it with a hefty support plan for a mission critical operation.

Sounds good to me! Establish a solid brand. Sell "systems" and "the experience" instead of hardware or software (think Apple). Provide an abundance of value without squeezing pennies from your customers - this also acts as free advertising. We'll see to what extent a certain industrial software company with killer products can follow suit.

Enterprise HMI Systems: Is the cost too high?

Another article by Steve Hechtman. He takes an interesting approach at the role of the industrial integrator in Enterprise Integration.

Modern Manufacturing

One hears so much about Enterprise HMI and Manufacturing Execution Systems (MES) that it's easy to suppose these systems are everywhere. A visit to almost any manufacturing facility will show the contrary. A recent article in InTech magazine confirms this with a discussion about “the promise of tomorrow's systems” as having “a global view of all processes within the plant via the corporate intranet/ portal ...” Why is this a promise of tomorrow? Does the IT sector not have the tools and standards to accomplish this right now?

Let's get down to basics. Production orders need to go to the production floor and production results need to come back to management. The information flow should be instantaneous so that real time corrections can be made. Relevant historical information should be available to management at all times. When these mechanics are in place management can do its job. Without them, management is likely to be loose and error prone at the plant floor level. 

The real question here is why, when the rest of the world is getting along with inexpensive and standardized IT tools, do plant floor islands of information still exist?   The technology is there and the need is recognized. So why the disconnection? 

Square Peg in a Round Hole

Most control system manufacturers have implemented and defended highly proprietary protocols in their controllers in an effort to lock-in end-users. This has been the game since controllers first hit the plant floor and the inevitable result has been high prices due to lack of competition.   So it's not surprising HMI systems offered by these same vendors are priced an order of magnitude higher than comparable IT technologies. One thing that all HMI vendors seem to agree on is the pricing model and none dare violate it. 

It's doubtful though that price alone would discourage broad deployment of Enterprise HMI systems since in a production environment the ROI should easily pay for the systems. But the more likely reason for lack of full adoption is that such systems are out of tune with mainstream technology. It is certainly evident that IT departments are resistant to deploying and maintaining these systems on their networks. 

Two Different Worlds

The manufacturing sector is about five years behind where it should be when it comes to integrating enterprise to the plant floor. Nearly every other part of the manufacturing enterprise has been integrated with the latest technology and this has pushed right down almost to the plant floor where it stops short. 

We have a picture where controls manufacturers seeking new profit opportunities have pressed up into the IT arena but their offerings aren't standard, are overly complex, are difficult to maintain and the pricing model unduly discourages full realization of what these systems could be . Conversely, IT departments and ERP vendors have tried to press down into the plant floor but this usually fails because plant floor control systems are highly specialized and aren't well understood. A lack of knowledge in this area can be a dangerous thing. A humorous example is when one IT department decided to “ping” all of the PLCs in a plant and automatically re-download non-responders. These are two different worlds and neither is likely to deliver a workable solution. 

The Controls System Integrator is the Key

Today, with OPC standards and with inexpensive OPC servers for practically any control system available, the only thing preventing successful integration of control systems and enterprise systems is a lack of knowledge. The control system integrator is in the fortuitous position of already having controls knowledge (the most difficult part) and only needs to arm himself with IT knowledge. A controls system integrator will find this technology rather easy to master. He might be surprised at the support and assistance IT departments will extend to him now that he is dealing in “their technology.” 

The controls system integrator is well poised to be the fence mender, the gap bridger and the builder of workable Enterprise HMI and MES systems for at least the next ten years.

***

This article was written and provided by Steven A. Hechtman, President of Calmetrics Company.  Calmetrics is a control system integrator who specialize in efficiency issues and the development of simple, powerful, connected systems.  They offer the following services:

• Design and programming of PLC control systems.


• Connecting PLCs to Powerful WebApps and databases.


• Producing clear and concise control system documentation.


• Emergency troubleshooting and support.


• Delivering PLC control and troubleshooting training.


• Advising on potential areas for efficiency improvement.

For more information on Calmetrics, please visit www.calmetrics.com

Web-based HMI: An emerging trend?

I just dug up an article on web based HMIs by Steve Hechtman from automation.com. It is a little dated, but still very relevant. I was even quoted in it :-).

Realization of Web-based Control

Once considered impractical for applications requiring responsive animation and real-time control, a new breed of web-based HMI system is starting to appear on plant floors and in manufacturing enterprises. “Java (web) based systems can now deliver sub-second response, rich animation and natural integration with other parts of the corporate information infrastructure,” touts Nathan Boeger of Inductive Automation. Unlike traditional systems, these web-based systems can economically be extended to every aspect of a business such as QC, maintenance, logistics,  plant manager, and so forth. Now every participant in the manufacturing cycle can have unprecedented access to vital plant production information.

It's easy to see why web-based systems are gaining popularity. Web-based systems install and run client applications from any web-browser and when users login they always get the most recent version of an application.   There are no client licenses manage, no tedious software installations, no application files to copy over and no communication configurations to setup. IT departments are willing to embrace technology they understand. All this is in sharp contrast to traditional systems. The economic advantages of using web-based systems are compelling. The bottom line is, web-based HMIs systems fit well with the rest of the enterprise and facilitate the smooth flow of information throughout an organization without unnecessary difficulty and expense. 

Security Issues

When potential users first consider using web-based technology they usually ask about security.   Just how secure are web-based systems?   The question is especially valid now that post 9/11 committees have deemed HMI and SCADA security “one of the most serious risks to our national security.”  Traditional vendors rely heavily on “security by obfuscation” which has never been considered a safe practice.  Web-based systems, on the other hand, are already positioned to leverage standard and proven web security techniques as administered by IT departments.

It’s only be a matter of time before legislation mandating minimum HMI and SCADA security requirements will surface. Traditional providers will likely have to overhaul their products to come into compliance. They will welcome this day since they will sell lots of mandated security upgrades. 

Seeing What's Next

Functionally speaking, HMIs haven't changed much over the past five years. “HMIs that just do operator interface tasks are a commodity, and you can buy them dirt cheap off the Internet…The real action is in HMIs that provide web access, interface to higher-level enterprise software, perform MES functions”, says Rich Merritt, Senior Technical Editor of Control Global, in his article, “HMI Software is disappearing”.

The book Seeing What’s Next  by Christensen, Anthony and Roth, introduces theories to predict major industry changes. These theories are supported with interesting historical examples. Applying these  predictive theories to this industry suggests incumbent HMI vendors will continue to service their large existing market without much change. They will probably not compete with their own model. On the other hand, web-based vendors will find success selling where traditional vendors have failed; to those companies who refuse to spend big bucks on systems perceived as being unnecessarily complex, cumbersome and overshooting needs. This is likely to lead to explosive growth for web-based systems in market segments which have been unfulfilled by traditional systems.  

Anyone familiar with manufacturing knows the majority of factories barely implement information technology at the plant floor level. There are exceptions, but when you see clipboards being used to record schedules, downtime and production, when you envision how things should be done, you finally come to realize this is a vast untapped market. 

There is an accelerating pace of web-based systems being installed in what was essentially a non-consuming market. Users are finally getting what they want – the functionality of an HMI with the economics of a web browser. The real question is not whether web based control systems are an emerging trend – they cannot be stopped, but rather which vendors are poised to jump on the bandwagon and deliver the technology.

***

This article was written and provided by Steven A. Hechtman, President of Calmetrics Company.  Calmetrics is a control system integrator who specialize in efficiency issues and the development of simple, powerful, connected systems.  They offer the following services:

• Design and programming of PLC control systems.


• Connecting PLCs to Powerful WebApps and databases.


• Producing clear and concise control system documentation.


• Emergency troubleshooting and support.


• Delivering PLC control and troubleshooting training.


• Advising on potential areas for efficiency improvement.

For more information on Calmetrics, please visit www.calmetrics.com

SCADA 2.0 - Present and Future: SCADA on the Web

SCADA 2.0 - Present and Future: SCADA on the Web

edit - Don't know how it took me over 2 years to find this blog. His early posts are particularly insightful:

http://scada2.blogspot.com/2005/08/so-what-is-scada-anyway.html
http://scada2.blogspot.com/2005/08/distributed-scada.html
http://scada2.blogspot.com/2005/08/opc-promise-of-connectivity.html

Sunny day for MySQL

Forgive the crappy title - and the plagiarized ideas. It seems everyone's recycling the same info. Sun recently announced plans to acquire MySQL for nearly $1 Billion. The companies should work well together. Sun is the biggest proponent of open source software - think: Open Office, Java, Netbeans, and Open Solaris.

I don't understand the economics or motivation behind the move. Does Sun intend on shifting the demographics of the server side scripted web site scene from the, oh-so-popular LAMP (Linux, Apache, MySQL, PHP) to some form of SAMP? Solaris was the first platform that MySQL ran on. Perhaps Sun is making a move at the mighty Oracle. Historically, supporting open source was justified as a means to promote hardware platforms. It certainly makes them the "good guy" in my book. Look how far that philosophy carried Google ;-).

The real question is how this affects the little guys. It's premature to make specific predictions. My gut feeling is that end users will benefit. Sun knows Open Source. They're committed to maintain the same "mission", staff, and direction. Hopefully they throw development resources to MySQL and make the platform more capable.

This looks to be a win-win for users. It makes me nervous that it's sitting on the shoulders of a baffling billion dollar move. Am I missing the obvious? It screams sellout to me - yet I can't think of a better company for MySQL to sell out to than Sun. So, I remain optimistic that this acquisition will drive a better, free product for all.

Web based SCADA - what's the big idea?

My control system isn't on the Internet! Why should I care about web based HMI or SCADA systems? I've been discussing these points for 5 years now, which is at least 10 years behind the corporate world. My ultimate goal is to educate end users on the pros and cons of a technology that may be very relevant to them.

"You'll sail upwind by a fire under your deck - I don't have time for such nonsense", my recollection of a quote from a Civilization video game. The point is that the application of new technologies isn't always immediately obvious. It helps to take a step back to consider what you're trying to accomplish.

Let's break down your SCADA system. Visibility into your process is probably the single most important aspect, which may come in the form of numbers, pretty graphics, etc. You need to be able to see the realtime and historical status of your operation. Next comes control. The ability to adjust setpoints, start and stop things, etc. After that it probably branches into: data input, reporting, alarming, interfacing with other systems, specialized applications, etc.

Sound like things a stand alone computer software package can do? Yes, that's an ideal version of the HMI that you're used to. Sound like something an HTML web page can do? Absolutely not! The first thing to realize is that "web based" is referring to a technology set that's prevalent on the Internet.

Now think of all the cool animated web sites you've been to. Java, Flash, ActiveX, DHTML and AJAX, and other technologies enable rich multimedia content. The big advantage is that the framework exists to run such applications with only a web browser. Instead of dragging CDs along to each HMI seat to install the software, then necessarily configuring: projects, data connections, PLC connections, server connections, etc, a new "web based" client need only the URL and permissions to access the system.

The next advantage has to do with routing and existing infastructure. How easy is it to connect a new computer on the other side of the plant to your legacy (non-ethernet) PLC network? It can get pretty tough! You probably already have an IP network throughout your plant to ride. What about realtime connections to your other locations? Again, our existing transport infastructure simplifies things with a "web based" approach. Does this mean that it has to be on the Internet? No, but it can. Or you can establish secure tunnels over a public network (Internet).

OK - so a "web based" approach brings nearly the same capabilities to the table, and can be easily launched from anywhere without the "traditional" lengthy installation and configuration process. What about safety, responsiveness, stability, and security?

It's true that the Industrial arena has considerations that don't exist elsewhere. This is why it's important to deal with people and companies that specialize in your trade. However, dealing with proven standards and technologies on the scale of decades of development and millions of users makes more sense than blindly trusting the "big" industrial software companies homebrewed nightmares. They are small as software companies go, and began with single terminal requirements that were much simpler. It's in their best interst to stretch their legacy junk in favor of backward compatibility, but even they are slowly turning to standards based approaches.

Remember that the Internet didn't grow out of the necessity to email grandma - it was the evolution of a hardened system designed for C2 (command and control), specifically launching missiles. Keep in mind that standard encryption based security schemes keep your investments safe from hackers. Web based SCADA provides a very capable, centrally managable, scalable and secure, IT supportable system. Wow!