VCP at last, and virtualization in Industry

I had the opportunity to attend VMWare's Fast Track course last December. It was a good class. The info came fast and furious! After nearly allowing my test voucher to expire, I buckled down and hit the books.On Wednesday I passed the test, so now I'm officially a VMWare Certified Professional!

Virtualization has been an interest of mine for awhile. I live by VMWare Workstation on my desktop and await the day when I get a laptop that's powerful enough to support my too-many VMs at once habit. For the Industrial Integrator/programmer/professional, Virtualization makes too much sense! Check it out if you haven't already! It readily solves problems that Industrial Software vendors have created (you need this version of Windows without a patch, can't run with competing software, older programming tools for some customers, etc, etc). Using multiple Virtual Machines, you can have exactly the right tool for whatever task at hand - without ever needing to make any changes. You get additional benefits of features like "snapshots" and "cloning". The only real issue is that none of the vendors have decided to support PCMCIA/PC Cards, so you'll have to use serial or USB connectors. This should matter less and less as time goes on.

On the server side (which is what the VCP is all about), VMWare may or may not make sense for your Industrial applications. Don't get me wrong - ESX and ESXi are awesome packages, but if you don't have a competent IT department - you're adding another layer of complexity for what might benefit you. For those larger companies that provide a lot of server based services - you're probably already using a virtualized environment. It just doesn't make sense not to.

Web based SCADA - a perspective from the Netherlands

Web based SCADA article by Eduard van Loenen of Yokogawa’s Global SCADA Center in The Netherlands. Kudos to Dan for pointing it out. He brings many great points to the table of the advantages to the trend of web based SCADA systems. 

From a security perspective, I disagree that IT technologies are "more susceptable to cyber attacks" than "rigid proprietary software". The cyber attacks will come since systems are being networked and remotely accessbile, which is a productivity enhancing user requirement in modern manufacturing. Proprietary software tends to be less secure than modern IT technologies, especially as the Internet makes exploits more widely available.

Good article, though!

Databases – The Perfect Complement to PLCs, by Steve Hechtman

Interesting insight on the role of SQL and databases with PLCs and industrial automation.

Hosted at Automation.com,  Reliable Plant, FoodEngineering

Ignition, post release activity and OPC vendors

I was impressed with the Ignition by Inductive Automation product release webinar. The Inductive Automation team did a first class job introducing the company, Ignition product platform and architecture, and the motivation behind the shift to OPC-UA. In a nutshell, their vision of the future of industrial software is: standards based, IT friendly, and platform independent. Get DCOM out of here! We want a top notch, secure SCADA system that runs equally well on Linux, a mac, or any flavor/service pack of the Windows rainbow. The demonstration included a full server installation, which took about 2 minutes, and highlighted the drag and drop ease of the new SQLTags History feature, which is deceptively simple, yet powerful.

We've been getting a huge number of hits to the new website by a number of companies across the world. We've also been getting a lot of interest in when more drivers will be available, like Siemens, Modbus, and Automation Direct in addition to the existing Allen Bradley suite.  I suppose this isn't surprising considering that we're the only cross-platform OPC-UA server available. While it shouldn't be too long before we get those drivers developed, this brings me to my next point...

Where are the other OPC-UA servers? We're really looking forward to a more vibrant marketplace for OPC-UA products. Kepware says they'll be releasing their OPC-UA server soon - we're looking forward to being able to test it with Ignition. This really excites me because it brings their extensive PLC product line to the table via OPC-UA. Unfortunately, I don't think it will be cross platform.

So, how about it? OPC vendors: the gauntlet has been thrown. The industry wants OPC-UA products! And if they're cross platform, all the better. When will we start seeing them?

My thoughts on securing your plant systems

Security, not your first concern as an industrial integrator or plant manager. Availability ranks numero uno. Got it! Your company loses $30k/hr during plant downtime. Got it! Plant workers are not exactly NSA hacker material. Got it! So why is security important? Look at those reasons again and above all, safety. Consider that your legal liability in court costs more than downtime. Do you want a disgruntled employee to shut down the factory on your watch? It is incumbent upon you as a professional to demonstrate due care. We're not worrying about international hacker rings or bored young college geniuses. Accidents occur and insider attacks are possible.

It's 2010 - many systems end up touching the Internet or outside world somehow, whether you like it or not. Common practices in the 90s, like allowing users to share passwords are unacceptable - that audit trail is a must. Having PLCs on the same network as office computers - professionally irresponsible. Your company uses 25 year old hardware that was never designed for security - it's up to you to isolate that network. Provide access with a "hardened" dual-homed (2 network cards) computer that is patched and protected by a firewall. Utilize VPNs, DMZs, VLANs, SSL, IDSs, and anything else in your IT department's arsenal. They know how to secure a network - it's their job. Gone are the days of operating behind their back, not letting them touch anything for fear that updates will break your system. Pick a vendor that IT will support. Practice Defense in Depth. Let them help you. It's 2010. Embrace positive change!

Introducing Ignition by Inductive Automation - Web-based, cross-platform SCADA Webinar

From an official Inductive Automation email....


Say goodbye to rationing clients, patchwork control systems and project development constraints caused by oppressive licensing systems.

Ignition by Inductive Automation is being released January 27th. Ignition is a major next step in the evolution of FactorySQL and FactoryPMI.

Join us for this new release webinar to see how the Ignition platform opens up possibilities in the industrial automation industry. The webinar will give an overview and demonstration of Ignition's features and its capabilities, such as:

• Web-based clients that enable scalability
• Unlimited clients and tags, licensed by the server
• A flexible database-centered architecture
• OPC-UA support
• 100% cross platform: Windows and Linux
• Simplified software setup and ...

Also accompanying the product launch is Inductive Automation's new website design at http://www.inductiveautomation.com/, which will launch January 27th with details about Ignition.

Register here to attend the new release webinar.

Title: Introducing Ignition by Inductive Automation - Web-based, cross-platform SCADA
Date: Thursday, January 28, 2010
Time: 9:00 AM - 10:00 AM PST

After registering you will receive a confirmation email containing information about joining the Webinar.

System Requirements
PC-based attendees
Required: Windows® 2000, XP Home, XP Pro, 2003 Server, Vista
Macintosh®-based attendeesRequired: Mac OS® X 10.4 (Tiger®) or newer

Space is limited.
Reserve your Webinar seat now at: https://www2.gotomeeting.com/register/273245266

Ignition preview and a visit to Inductive Automation

Over the holidays I had the opportunity to visit my good friends, Colby and Carl, who are lead developers at Inductive Automation. They gave me an in depth technical "sneak peak" of what is becoming their next product line, "Ignition".

The plan is a single product platform that replaces the separate functionality of FactorySQL and FactoryPMI. They really went all out with levering technologies! The product is platform independent (Windows 32 or 64 bit, Linux, Mac, etc), fully web based (no client installation), and supports separate "modules" that all have access to the same powerful core technologies. Everything runs from a cluster aware web server similar to the current FactoryPMI Gateway but they had new behind the scene tricks (Non-Blocking IO for multithreading, advanced serialization schemes for saving/storing projects, and a whole slew of security upgrades consistent with the OPC-UA specification).

They showed me an OPC-UA application that securely connected to different PLCs (many AB, Siemens, and AD) via different PCs without an OPC tunneller. All this worked equally well on Windows and Linux systems. Once the data is accessible from the PLC, you get the full visualization spectrum that FactoryPMI already offers (graphics, multimedia, historical trends, reporting, etc).

The coolest part is that development time should be cut down substantially, for both Inductive Automation proper and 3rd parties/end users as the API is released, since the platform performs so much native functionality (network, database, project launching/hosting/editing, clustering, storage, alerting, auditing, authentication, etc, etc.). The Ignition platform should become to controls software what Metasploit is to ethical hackers/penetration testers.

OPC UA info

I just ran into this article dated May of last year. Simone Massaro of Iconics describes the direction that they went with OPC UA development. It's a bit technical, but a good read.

http://www.plantengineering.com/article/talkback/183110-What_is_OPC_UA_and_how_does_it_affect_your_world_.php

Is anyone buying this? Really...?

I stumbled across this article that, quite honestly, at first pissed me off. After a little reflection, I can only laugh. It reminds me of someone trying too hard to sell those $150, short gold plated digital Monster audio cables - Oxygen free or whatever. (If the engineer inside you doesn't laugh, then cry for the sake of the suckers, read on).

Maybe I read in to far, but I see snakeskin oil vendors grasping for air! The piece is an obvious response to Steve Hechtman's very different article on the same topic (hosted on Control Engineering) - he should be flattered. You see, the big vendors, GE Faunc in this case, but the exact same applies to Wonderware and Rockwell have been long committed to the concept of Historians, a glorified and expensive datalogger that includes, and is only meant to work with, a custom version of Microsoft SQL Server (of all product choices...). The problem is that now much cheaper products from companies like Inductive Automation and Software Toolbox can do a better job using any RDBMS (database) system. Being vendor-neutral, inexpensive add-on packages also do much better for things like trending, reporting, and data analysis. The biggest mistake of the current generation of Historian is that they tried to implement and include everything themselves - like making a giant Swiss Army knife with a spork, usb memory stick, and a wine glass. Now they're caught with their pants down, desperately scrambling to recovery their enourmous sunk costs (my favorite business term).

• You're data is special and requires "plantwide historian" treatment...their example query “What was today’s hourly unit production average compared to where it was a year ago or two years ago?”. I won't even comment...

• Your database needs to speak specialized industrial protocols (OPC) - There's separation of function by design and for a reason. Besides - this doesn't even make sense.
• Faster speeds and higher data compression - no way! The historian is wasting CPU cycles in both directions, which obfuscates your data (can no longer use external applications), to do something better achieved by a RDBMS system that supports it.
• Robust redundancy for high availability - is this a joke? Maybe Amazon.com should migrate their server farms over to GE-flavor SQL Server.
• Enhanced data security - another losing battle for the historian. The white paper mentions SQL injection attacks - all platforms in question can use stored procedures, and are all subject to this sort of attack. When it comes to up to date patching, arguably the most common vulnerability, SCADA vendors have the absolute worst track record! IT keeps their servers patched as a matter of practice - they're typically afraid to touch the SCADA machines. Ultimately, the "do everything" approach provides many attack vectors.
  

I can't blame them for playing their hand. I just wonder - will anyone read this white paper and take it at face value?

SCADA security and cyberspace threats

I probably sound like a broken record by now, but SCADA security is not going anywhere. This applies to almost anything electronic that is connected to a network.

Due to the nature of the systems, and the fact that they most often can't be easily patched, it's becoming increasingly important to choose standards based products/technology and protect your network infastructure. It's imperative to mitigate risk where you can!

In recent news - military arming for cyber warfare. This blog post commenting on hacking infastructure.

Matrikon and Wurldtech Cooperative

Wurldtech Security Technologies has committed to apply their Achilles testing technology and certification methodology to Matrikon's OPC products. Successful completion will place the Matrikon OPC Tunneller and servers at the top of my recommendation list. This is a big plus for the world of SCADA security! Now, if only we could do something about our legacy systems...

Wurldtech blog announcement and press release.

The Risks Digest

I stumbled upon the "Risks Digest - Forum On Risks To The Public In Computers And Related Systems" from another blog post. Some of the stories certainly made me laugh - I spent way too much time there!

An interesting conversation with a traveller in Bali

I had an interesting poolside conversation with "Jeff" at a resort in Bali. He works for Juniper networks, setting up core networks for huge accounts overseas. He told me of the $400 million project in Malaysia that will go on for the next 3 years, and about his project in Brazil. He said that he's gotten used to the long flight back that he takes monthly, but hey, how bad could business class be? More importantly he mentioned that Cisco's IOS is outdated - that engineers left there after being turned away with (then) cutting edge ideas of using ASICS (specialized integrated circuits as opposed to generalized processors) in routers. They maintain a monsterous market share and provide lots of enterprise services (like voice and conferencing over IP). It was weird to hear him refer to enterprise accounts as the small ones (compared to major telecoms and infastructure).

Poking into Jeff's past revealed a masters degree in Electrical Engineering, a CCIE certification about 10 years ago (distinguish youself - don't mess around with the little ones, he said), and "various others". I guess he did some defense contracting at the Pentagon earlier as well. But I got the usual, "certs and education get you in the door...specialize and learn the industry to move up" explanation.

So why mention any of this here? It's all about infrastructure! First, it used to surprise me that I get a 100 megabit Internet connection at home in Korea. That's not fast in terms of network equipment and Korea is on the cutting edge. Heck, I'm getting 700k at the resort in Bali! He was talking about OC768 (40 gigabit) core routing equipment in Malaysia! There's plenty of fiber under the ocean! We, the US, have piles of legacy equipment that we're dealing with. These new countries coming online get to engineer their solutions properly and deal with the latest and greatest! Mr. Obama - if you're reading this, I think investing in our digital infastructure would make a great part of the stimulus package! Our industrial control networks will benefit from such upgrades.

Getting the most out of your SCADA system

I literally visited Inductive Automation the day before Gary Mintchell did. I didn't get the opportunity to meet him, but I did get a glimpse of his insight. He quoted the company as a "database company" - as a foundation, which is an insightful perspective.

Here's what occurred to me - I've been involved in big projects and small projects, private sector, government, and military running a variety of platforms. Does anybody have issues creating a tinker toy HMI with a few setpoints and graphics that change color? I really doubt it. Which vendor would I recommend for that? Who cares - they all do it. That's what Walt refers to about the commodization of HMIs.

So what's valuable and where are we failing? My top choice, and they are together, are "customizability and interoperability", something we tend to suck at. Suppose I asked, "how much power have we used so far this month". I'd likely get, "I donno - but you can figure it out if you keep a log of readings from that meter". Or - "what's the status on that shipment we sent out last week"? It's available on the Fedex/USPS web site. But why not on our information/SCADA system? Isn't that what web services is all about? Who are we kidding - we have enough issues migrating/tying in our legacy systems. I bundled "customizability and interoperability" together because the point is to be able to tie your system to others easily. Managers shouldn't have to buy hardware and large amounts of integration services to make their systems work for them.

Great! So how does this work? The key is being able to pass data - through standardization. This is where technologies like OPC (UA) and web services come in. But another huge, often overlooked method is using SQL databases. Most applications, and nearly all business systems use them natively. You want to know anything about your process - inventory, QA, for example - past or present. That should be available in your SCADA system. It's a great connection point, provided that it's flexible, which is Inductive Automation's strength. Get that SCADA vendors - hint, hint - step away from the custom Microsoft SQL Server implementations! The royalties are great, but nobody belives that you need them for performance. Besides databases are useful for other reasons than being a historian! It's not hard to support Oracle, MySQL, DB2 and others - just swallow your pride and old company lines.

How do you get your existing or legacy system to interoperate with others? Simple, OPC <-> SQL database bridges exist for that purpose.

OPC Interoperability Conference, UA and Java

In the spirit of catching up with my backlogged blogging (recent personal Japan trip from January), I'll post about a few topics that I missed.

I had a chance to visit old friends at Inductive Automation. They gave me a demonstration of the working Java OPC UA stack that they unveiled back in the beginning of March, at the North American OPC Interoperability conference. The "test program" was a slick AJAX web page that browsed, read, and wrote tags to an AB SLC with no noticable delay.

The Java UA stack is significant for a number of reasons. First, the UA spec is notional. I'd guess that the OPC Foundation hoped, but didn't really expect, to see it implemented independently - at least not right off. (*a Java stack on their C/C++ implementation is planned with a pure jave stack in the dreamy future) - (*correction again - Randy Armstrong points out in a comment that a Java stack is currently available). This leads to the second point about Java being platform and Operating System independent - everything supports the Java Virtual Machine these days. The point is that we have millions of users across continents and lots of reasons to seek Windows alternatives. I'd bet that there's a dissociated army of programmers in the industrial space who are doing their own thing, but would jump on a standards based bandwagon. That's really what our industry needs for: efficiency, simplicity, and cost savings. The idea being that everything "speaks OPC UA" so historically dissimilar hardware, appliances, and applications can talk with ease - securely.

Which brings me to something I heard about at the conference. Reportedly, the UA guys were asked to go home the first day so that all the legacy apps could be set up. This makes me laugh and wince simultaniously! It's not uncommon for a room full of experts to spend an afternoon getting two nodes to talk to each other - it's all about Windows DCOM security, which is equally painful as it is full of gaping vulnerabilities. At the point where you're communicating with a friend, a third party can't see either.

New standards are a funny thing - everyone knows they're coming, everyone knows they'll benefit from them, but you're not ready to commit until the next guy has. Kudos to Inductive Automation for getting the ball rolling. Kudos to Kepware and Iconics for the same. Siemens has comitted to an entire product line! Wonderware's been talking the talk, as has Rockwell (both in 2006). Here's to them coding away in their secret labs! Don't believe me - here's a video of how great and mature OPC UA really is, complements of Eric Murphy of Matrikon! It's a riot - I promise :)!

Using Open Standards in Water and Wastewater

I hadn't expected to be blown away by Inductive Automation's Using Open Standards, web-based modern SCADA technology to manage your water operations webinar, but the collection of speakers and content was phonomenial!

Don Pearson, the moderator, opened up with a brief presentation on federal 'relief' monies specific to water and wastewater. Henry Palechek presented trends that he's taken Helix Water District of San Diego through from their $2 mil VAX system, to their $350k Wonderware system, to their existing FactoryPMI system. He had a lot of interesting insight into the decision process and business ramifications of his choices. The transcript can be read here (you can also listen to a recording of the interview from the webinar).

Patrick Callaghan of MCS Integrations then presented a system integrators prospective. He showed off a live system (which, by the way, made me unnecessarily nervous) that he wrote for the City of Largo Vista in Texas. His setup was INCREDIBLE!!! Operators run around with tablet PCs, connected via VPN over the cell network. Everything looked sharp and screens were linked together intuitively. He had screens where he could create groups of operators on the fly that recieve alarms at different intervals until they're acknowledged. He showed generic tanks and valves that would display different values based on their types, but used the same objects/windows. This included animation based on setpoints (levels in a tank), an alarm history, trends, and even a custom note field that would record the operator/date_time and create an overlay icon on the main screen that showed the existence of a note. The trend screen allowed you to create, save, and edit arbitrary groups of pens. He had all sorts of reports autogenerated that operators needed to generate, and had an integrated pdf library of the ones they had to fill out by hand. It's hard to fully describe how sweet his SCADA package is - it seemed more to me like the product of a $1mil well written custom application that was tailored to the client. I've never seen such a thing from Wonderware, Rockwell, GE, or the others.

That webinar was fantastic! I'll keep my eyes open for more.

Cheating in Online Poker

I just got back from a Las Vegas weekend trip. The poker gods were good on the cheapest ($1/2) tables Bellagio had to offer. My hourly return wasn't impressive, but I had a great time chatting with a variety of gamblers.

One particular story clung. A player said that his "friend" got a chance to witness someone "win" $13k in one night cheating at online poker. I'm not too impressed with the usual tactics, run a background application to gather statistics on opponents or even the recent Absolute Poker cheating scandal. This one caught my attention because it was so simple, yet incomprehensible to catch. You could multiply the benefit with automation/a program, but that's not necessary.

The scam involved playing 5 of 6 simultanious accounts at an online poker table. It shouldn't take a superstar to see that you could easily squeeze out single unsuspecting victims. You could even use a program to obtain more accurate odds since you see 1/5 of the deck. Connection details seemed obvious. I would use proxy services to route via different cities around the world, consistent from each account. The crux of the scam lies in the fact that you can easily create throw away identities - violating the security pricipal of integrity, that you can verify that someone is who they claim. Online gaming sites do monitor IP addresses (defeated with proxies) and users who constantly collaborate. However, you'd be pretty hard to spot with a pool of accounts that get used for short time periods.

What about the penalty if you do get caught? I can't imagine playing multiple online poker accounts getting you in as much trouble as stealing...

The best protection brings inconvenience - closely couple user accounts with real people. That requires you to give up all the personnal info that you don't want to share: valid ID, bank accounts, SSNs, etc. As an online player I'd feel much more safe if the site required heavy verification. Then again, I only play online for "points".

Doesn't Cyber Security Deserve a Stimulus? - Wurldtech

A perspective worth reading - http://www.wurldtech.com/blog/?p=119

And another - Walt, I don't agree with every detail, but the message is spot on!

http://www.controlglobal.com/articles/2005/397.html

CISSP at last!

I've had Shon Harris' All-in-One CISSP book on my shelf for years. I've taken it on long drives and flights without so much as cracking the pages of that great volume. It finally took a week long class and the discipline to study before I was ready to commit. I took the six hour plunge in December and recently found out that I'm officially a "Certified Information System Security Professional". Yay!

The real significance is embodied by standards organazations and my new professional community. It's all about the articles, networking, and even the new forum that I already spend too much time on. My goal is to continue to push good security practice in the Industrial (SCADA, HMI, controls) space.

Opinion: Do you need a $60,000 process historian to log data? (Control Engineering)

Interesting article featured on Control Engineering

-- Control Engineering, 3/26/2009
Steve Hechtman, Inductive Automation

I wish to register a complaint. There is a rumor that has been circulating for years that relational databases are too slow for fast process data and that only process historians are up to the job. Vendors of process historians will cite sluggish performance and the lack of data compression as the reasons standard off-the-shelf relational databases won’t work. Apparently the last time they used a SQL relational database was a few decades ago.

While there may be some specialized domains where process historians have a niche, they are not a practical choice for most industrial applications. In effect, historian vendors are saying your Toyota Camry is inappropriate transportation because it is incapable of going 180 mph or finishing the quarter mile in under 10 seconds.

The rumor denigrating relational databases for poor throughput is baseless. A standard, off-the-shelf Microsoft SQL Server coupled with FactorySQL can log in excess of 100,000 tags per second using a desktop machine. In all likelihood, other factors such as the industrial network would become bottlenecks before the database does. Furthermore, today’s generation of SQL relational databases are designed to scale gracefully to power high-volume Website traffic, whose load peaks dwarf those of industrial controls applications.

Data compression is an area where process historians do score a point. However, even this consideration can be handled with standard off-the-shelf SQL relational databases. Take a look at the MySQL 5.0 Archive Storage Engine which achieves on average a four to one compression ratio. Proprietary process historians may beat that, but let’s get back to the point of practicality. Hard disk space is so cheap these days that even considering this point is becoming an anachronism. For the rare application that demands it, table compression coupled with intelligent data logging allow databases to compete even in this regard.

One crucial question that process historian vendors omit is: what are IT departments willing to support? When I make initial contact with IT folks, I always ask which relational database they use. Then I assure them we’ll work with that. This generally makes them very happy. Believe me, you want IT on your side or your project will end up on a data island which is useless in an enterprise system. Think of it from their point view; they have the training and tools, generally, to support just one type of database. With these tools and training they can support the database with scheduled backups, tuning and other maintenance.

Okay, we’ve heard process historian rants about relational databases; let’s talk about the downside of process historians. Let’s start with support. Just check the Amazon bookstore for any one of the proprietary process historians and you’re likely to come up empty handed. On the other hand, check for “SQL configuration” and you’ll come up with hundreds of books. How about finding people to support these proprietary systems? Good luck.

Then there is the concern about supporting relational data with a process historian. Frankly, the middleware layer is all about relational data. Time-series data, which is what process historians deal with, is just a fraction of what is needed in the middleware layer. Correlating batches, shifts, inventory, orders, downtime, quality, etc., is purely relational in nature, and these are the features that today’s enterprise integration projects demand.

What about a cost comparison? The process historian is going to be ten to thirty times the cost of a relational database using a driver like FactorySQL depending on the number of tags required. The controls industry is still backwards on this point and prefers to price its software per tag as though the extra tags cost money to manufacture.

In summary, we’re talking about practical choices. The Ferrari may be great fun, but do you need a $500,000 vehicle to drive the kids to school or would the Camry suffice? Likewise, do you need a $60,000 process historian to log data? A relational database makes a great historian, but the reverse isn’t true. A process historian cannot process relational data. For the vast majority of systems, a relational database has more than enough power to service the historical and relational data requirements, making it not just the practical, but the wise choice.