Business is booming for Wonderware

Wonderware hired another 200 employees this year for a total of around 500. I'm no expert, but this still seems "small" for a software company and big for "nitch" software. They're developing and marketing toward specific industries and growing worldwide. They're fighting for legitimacy in the "Enterprise", even while companies like their "partner" SAP tries to compete directly, and powering the olympics. They even recently started selling Panelview style hardware to run Intouch 10 on. Great job from the leading SCADA company.

Read more here.

OPC UA Coming to life!

Lots more OPC UA buzz - looks like it'll be really materializing soon - hopefully later this year. I personally can't wait! Remember, Iconics and Kepware already pledged to bring an end to end UA solution soon.

http://www.controleng.com/article/CA6559411.html?rssid=129

Edit - more contributions pointed out by Eric Murphy of Matrikon
http://www.ceasiamag.com/article-3936-theyearofopcua-LogisticsAsia.html
Automation.com OPC portal

Another SCADA security vulnerability

What a great day - another SCADA security vulnerability uncovered! I don't want to see any industrial software fail, but I know the nitch products were written in a vacuum without security in mind. HMI packages are buggy and susceptible to attack - particularly the older ones still in use! We need to get over it, confront the issue, and fix it! Hopefully vendors have the integrity to self-test and release patches on their own, but this won't solve the problem. A developer simply can't find all of his own bugs in a test environment. The more computer security research groups start looking into these "little" but significant applications the better.

This time it was Citect, Wonderware had the last one. SCADA vendors be ready - you're next! That's you: Rockwell, GE, and Inductive Automation!

FireFox Download Day announced

Tuesday, 17JUN is Download Day (FireFox 3 if you havn't been paying attention)! Check out a graphical world pledge map here.

Rockwell Automation selects Matrikon OPC for FactoryTalk Suite

Press release from Matrikon and Rockwell Automation submitted to automation.com.

It looks like Rockwell is finally learning not to try to reinvent the wheel with every product (ie homecooked remote DCOM via RSLinx Gateway) in favor of getting professional support from those that know best. Customers will be getting the benefit of OPC Tunneller (very important now for most remote cases, they could have already used but probably didn't know it, should become obsolete with OPC UA). They also get the benefit of various DCS, telemetry, and PLC drivers. I'm not sure what "IT" drivers are but it sounds good. New (and highly promoted) are building automation drivers such as: BACnet, Johnson Controls, and LonWorks, which is a welcome natural fit to HMI and SCADA systems.

I believe Rockwell has been working with Kepware's KepServer EX for RSView. Correct me if I'm wrong - I don't think it would help with programming tools (read RSLogix) or the FactoryTalk versions of their software (read RSView SE series).

It's nice to see more collaboration between industrial software companies.

http://www.automation.com/content/rockwell-automation-selects-matrikonopc-for-3rd-party-connectivity-to-their-factorytalk-suite

The significance of updates

Software updates - What a sticky topic! Do you pay? Are they free? How frequently should you install them? Should they be installed automatically? In some cases you find yourself vulnerable to exploits by not upgrading. In this case, you might "brick" your home router. Yes, we're talking about Windows XP Service Pack 3. Changes to the network stack manage to send Billion "BiPAC 5200-series routers to go into a constant crash and reboot cycle".

Here's my take. In general, it's good practice to stay updated. I'd hold off on critical systems, especially in closed environments (not on the Internet). For those cases you might consider a policy of first upgrading a test environment. At the very least, schedule upgrades during times that will minimize downtime impact. And - have a backup!

http://apcmag.com/router_crashes_blamed_on_windows_xp_sp3.htm

The power of the people

I find myself frequently trying to convince industrial professionals to be active in their communities! Besides trade shows, I'm referring to forums. Specifically, MrPLC and PLCTalk. If you're not registered, go there! If you're not active - post a few. Try it, it's addictive!

In this case a member got a little voting help for a free home makeover. His place looked pretty bad and he deserved it, but it was the PLCTalk folks that made it happen.

http://www.plctalk.net/qanda/showthread.php?t=38863

Building Automation and OPC

An AutomatedBuildings.com interview with Tom Burke of the OPC Foundation and Sean Leonard of Matrikon.

http://www.automatedbuildings.com/news/jun08/interviews/080527031642opc.htm

The interesting thing to me is how similar building automation is to industrial automation, yet how "stove piped" their respective products have been. Even the protocols, Leonard mentions "BACnet, Johnson Controls, and LonWorks", are totally different.

The interview rolls the grandiose OPC UA ambition into our sister industries, in addition to the direction you're used to, "up", namely Enterprise Integration. It focuses on OPC campaigning, providing interesting insight on what The Foundation is focusing on.

Good to see things moving forward. Early OPC UA buy in is going to be important.

Do your part!

Mozilla aims to set a one day world Download record with the official release of Firefox 3. The date isn't set, but you can sign up to pledge to download it on day one here. I for one, love Firefox and have been running the beta for some time now.

SCADA Security unnoticed

Core Security released an advisory on a Denial of Service attack where an unauthenticated user can take down a Wonderware system running Suitelink with a malformed packet. This was first brought to Wonderware's attention at the end of January. They acknowledged the bug by late March and provided a fix by late April. The advisory published in June.


Is it just me? A lot of people talk the talk about how important stability is with industrial control software. But nobody really cares. Trivial bugs (by comparison) get much more press. Could you imagine if a simple program could send a packet to crash a Windows server application? The press would go nuts! By Wonderware's account, they've sold 500 thousand copies that are running in 100,000 plants worldwide in virtually every industry. These plants are not all on closed networks! Good thing terrorists don't have access to Google, like I do.

Besides not wanting to share details with the public and not recognizing the problem in a timely manner, Wonderware did their part. It's a learning process that will hopefully go more smoothly next time. What astounds me is the fact that you don't see or hear about this except in a few very specific sites and blogs. I bet there will be a significant percentage of vulnerable systems several years from now - a combination of the weak promulgation of information and the reluctance of industrial users to upgrade unless forced. The latter caused by vendors releasing patches that haven't been adequately QA'd. This is one point where Inductive Automation is ahead of the power curve. Since FactorySQL and FactoryPMI upgrades nearly always come with free feature additions, IA users have created a culture of frequently upgrading their software.

But I digress. SCADA security is a huge bomb waiting to go off. There's a little talk on the subject, but the industry fails to take it seriously. I hope we can figure things out before the next 9/11 forces government intervention - how would you like your plant to operation like an airport? I'd like to think that level of regulation is unnecessary. We should each do our part in tightening down industrial security.

http://www.coresecurity.com/?action=item&id=2187
http://isc.sans.org/diary.html?storyid=4390

Inductive Automation Videos

Between traveling to New Zealand and Japan I've been pretty busy. I'll try to keep up with small blog posts. I've had lots of great ideas that haven't come to fruition.

I've been working on training videos for Inductive Automation. I'd love to hear your feedback. Once I finish a few more basic series I'll be taking requests.

Some OPC UA links

Great insight on the transition and future of OPC-UA from John Weber and Nathan Pocock of Software Toolbox. I hope he's wrong about the slow death of DCOM, but I wouldn't put a wager on it!

OPC-UA vendor perspective from Tom Burke of the OPC Foundation. It's a rather high level vision description of the architecture and commitment of the foundation. You're the man, Tom!

There are other good links. They came from here.

Linux obsession on the forums

Lately the usual PLC forums have been unusually chatty about Linux. Perhaps it's the recent Windows update that installed the new version of the .NET Frameworks 2.0, the trend toward Open Office and Star Office, or the successful home experimentation with Ubuntu and Fedora. There's also been more complaints than ever about the complicated and equally crappy DCOM basis of OPC, which gets (probably unfairly) pinned on Microsoft. The OPC Foundation gets to be the knight in shining armor with UA. General users are correctly getting the sense that configuration, specifically security, need not be complicated and that getting rid of the old also brings the freedom ditching the entire (Windows) platform.

The consensus seems to be that programming software will remain Windows based for some time. Until users put the pressure on - and they're complaining, but not applying diddly-squat, nothing will change in that area. So it's a 90% solution - control systems can be chosen on a platform independent decision, but integrators will still be running around with Windows laptops to program the PLCs. That doesn't bother me too much. First, they break the things every couple of years - about as often as Windows seems to fail. Second, there's always virtualization, which has been getting easier and cooler over time.

I welcome the simplicity of the newer Linux build frondends and their (always) powerful backend. I remember when the thrill of playing with a new Linux system came to an abrupt halt when you realized that your devices weren't supported and the user interface sucked! It's great to see "normal users" have such a positive experience with installing such a new, foreign, system. If only they knew the complexity of the innards! Thank god they don't - they're users.

A Red Team strikes!

Talk about an eye-opening experience! We invited a Red Team to do penetration testing on our network. A group of ubergeeks brought their laptops, tools, and toys and left with astonishing results! It's one thing to hear someone say that your server isn't up to date so "go patch it", but quite another to have someone steal a copy of your most valuable data in a short period of time, in a manner that's difficult to detect after they revealed how they did it!

The takeaway is that, if you expect to protect your data, you had better implement, maintain, and enforce Draconian policies on your network. Engineer the system from the ground up with security in mind. Implement IDS systems, baseline your network, track your user group memberships, and above all, train the heck out of all privileged users, then all users - they should be doing things like never logging in with their admin accounts, only using "run as" when necessary, like the Linux "su" mentality, and logging off of or locking their workstation when leaving for a moment. Keep updated with patches on all systems - servers, clients, printers, appliances, routers, whatever. Minimize where you can actually perform administrative tasks from. Apply your same password policy to everything that uses a password. Then: monitor, monitor, monitor. This is one of those "weakest link" deals. Powerful "server" applications like SQL databases (and many, many others) are great launching points - consider running them on "member servers" instead of domain controllers. Technically savy malicious users don't need much of a launching point. This doesn't even take "social engineering" into account, and users tend to behave stupidly from a security perspective.

After seeing what these guys did, I'd have a hard time imagining a network that wouldn't be vulnerable to that kind of crew - unless it's physically isolated, physically protected, and assured that malicious users can't get physically near it or trick actual users into inadvertently assisting them - an unrealistic network utopia. However, that said, there's a lot that you can do for network security. Don't look for a "magic bullet" - attack the low hanging fruit and take it from there.

VoIP - more than you bargained for?

Thanks for the inspiration Carl - your objective viewpoint often sheds light on absurdities that I might otherwise fail to notice. In this case I'm referring to Cisco flavored VoIP. It's an amazing concept that truly delivers next generation communication capabilities. Who wouldn't want: virtually unlimited directory numbers, multi-line capability, much more wide open conference call ability, a centralized web based interface to administer/log the system, the ability to tie into POTS lines at the gateway or use a truly cheap international carrier, and much more?

I had always thought that the key selling point to VoIP is that it works on your existing infrastructure. You already have Ethernet. You're connected to the Internet on a relatively wide pipe. Why not plug phones into your system?

What if I told you that my Cisco rep pointed out that they made half of their income - yes, literally half one recent year on VoIP related sales? Holy Cow! You know how much Cisco equipment costs! How much could a phone possibly run!?!? This surprised me even after hearing that they wanted nearly $20k for call manager software for one switch that we already owned. How could that be?

Lets jump to my recent VoIP experience. We rapidly deployed about 20 VoIP phones on a network that was also to become busier than usual. Without getting into the specifics too much, the system was very sensitive to being properly configured. If the separate VLAN wasn't set up just so phones would randomly reboot and exhibit strange behavior. Some of the switches with a slightly older version of IOS had to be configured differently than the newer ones. Phones plugged into switches without PoE needed power cubes. Gateways, call managers, and phones needed to be configured and coordinated between sites. Bottom line - I love what VoIP brings to the table and am eager to learn more, but it's not a simple matter of plugging a phone into your network.

OK, back to Cisco sales dollars. In light of what I just presented my earlier statements should make more sense. How would you like to save money by purchasing some new VoIP equipment that will work with your existing infrastructure? Yes, great. Well, you may need to replace all your switches. Update your routers, (maybe?), buy some call managers and gateways while you're at it. Hey - at least you'll have a modern network. Your cabling should still be good.

Given all that, I still can't say that I'm opposed to VoIP. In fact, I'll call it progress. But you sure as heck better figure out what you're getting yourself into prior to making the big plunge! You don't want to make yourself that guy at your organization!

The future is...Linux?

I just finished reading this year old article entitled, "Windows Vista, The best thing that ever happened to Linux?". Like many other pro-Linux (read anti-Microsoft) papers, it offers compelling arguments that I find myself agreeing with. In fact, my specific Vista complaints were very similar to theirs (all the hardware intensive eye candy that OS X does better, dropping important announced features: WinFS (relational database based file system), PowerShell (advanced scripting), SecurID (authentication for network resources), and PC to PC synchronization). They continue to enumerate Microsoft attrocities and go on to how Linux will dominate the future - think OLPC "One Child Per Laptop", the $100 PC.

I can't say what the future holds. I do know that every time I install Linux on one of my laptops (for anything other than when I used to program school projects), it ends up being too pesky to reasonably use. I get wireless networking/printing, DVD playback, a word processor, and everything else that I "need" straight - then never end up using it. But Linux always wins the theoretical argument - what could be better? I also know that well written Linux "appliances" work well. My M0n0wall router served me so well up until I bought a QoS enabled "gaming" router. We've been tossing around the idea of a CD bootable, lightweight Linux image designed to run a FactoryPMI client. Like Knoppix - maybe based on it.

In the end I always find myself going back to Microsoft - it's disgusting! They ultimately steal, buy, or reinvent the better technology, and it works well for them. Remember when SQL Server used to suck? I do - but SQL Server 2005 is a great product, thanks Sybase! They're getting into virtualization. It sucks now, but mark my words, they'll be giving VMWare a run for their money in a couple of years. What about Microsoft Office - do you really ever want to use anything else?

So this gets me to the recent official release of FactoryPMI Linux support. My first reaction was, "Who cares? You can already run FactoryPMI clients on Linux - they're Java based"! After thinking about it - it's the direction and comittment that matter. I align much better with Linux ideology, and the community is rapidly growing. Who else is tired of problems with every new major Microsoft release? Also, the Open Source community has a lot of software developed to bring to the table that's very powerful, but still a little rough around the edges. Microsoft makes good products that I like to use, but would rather not be stuck with. The percentage of HMI/SCADA vendors in bed with them makes me sick. Entrapment, not standardization becomes a predicament for end users.

In the end will it be Linux? Microsoft? Who knows - let the greater community decide. For me, the most useful products win. I'd like to see Microsoft include the power features that they've been advertising since 2004, and Linux to get progressively easier to use for all levels of user. Kudos to the companies that let users decide on what platform they prefer.

Industry news update

Quick summary of industry news, compliments of InTech Editor Greg Hale's Blog. Thanks for the great info!


Wonderware - Predicting growth in Africa. Still working on India, China and Eastern Europe, but Africa's where it's at in the next 30 years or sooner. Full entry here.

Kepware - Partnering with Oracle and adding an OPC Client to their flagship OPC Server KepServer Ex to all pass through support for 3rd party OPC servers including diagnostics. Exciting! Full entry here.

Rockwell/Cisco partnership - release Rockwell branded Cisco Catalyst switches that offer easier configuration, configuration via RSLogix 5000, and pre-canned support for common (Rockwell Centric) Industrial Networking protocols as CIP and Ethernet/IP (Industrial Protocol). It's about DAMNED TIME! I've always said that AB makes good PLCs. They're finally leveraging developed technology for industrial applications instead of reinventing the wheel over and over! That said, I'd fear the price tag! Full entry here.

Virtualization and SCADA, mini-SCADAs

Ever feel like a broken record? I get that feeling when "my" last 2 good post ideas came from following the crowd. Looking back, I haven't yet posted on SCADA security in response to the flurry of blog activity on the topic and the alleged "SCADA Internet attacks on the power grid" where the CIA keeps coming up - again and again. I've seen how the media quotes "the government", my 19 year old Seaman recruit sailor was "A Navy Spokesperson". The reporter was attractive - he didn't stand a chance.

Well, this post is supposed to be about Virtualization, an old topic in computing with renewed vigor! Other bloggers are talkin' about it, so why shouldn't I? The basic idea behind virtualization in this context is to work on logical hardware in a bit of a sandbox. Another nice feature is working from images (snapshots) instead of entire hard drives and machines. Imagine building your HMI exactly how you want, then taking a snapshot. With virtualization, you can run multiple instances of this. Your SCADA installation is an image file that can be run on any computer! Maybe you want to consolidate hardware, or maybe you want a similar environment for your QA department, or for development. The concept of "create once, use many" applies here.

Unless you're a software developer or running a computer lab, it's probably your servers that have the most to gain from virtualization. Servers are notorious for being resource underutilized, and are often fickle - how many of you would be comfortable "cutting over" most of the services that any one of your servers provide to another machine? You might not mind installing something new on a server, but I doubt that you nonchalantly move things around on production machines.

Let me paint a picture. You're starting a sizable new plant from scratch. You decide to buy a single $50k server from Dell as the main workhorse. It will be running "8 servers", (domain controller, database, web, email, etc) each with their own: memory, IP address(es), etc. Once the system is up, you decide that you don't want to install anything without testing it first. Fine, you use the same image on a different machine, install and test the software, then copy that image onto 'ol beefy. Suppose your email server needs more memory - you simply assign 4 gigs of the 32 total to that "instance" instead of 2. Now suppose that you're supporting something that's architecturally heavy, like Wonderware or RSView via thin clients on a Terminal Server. All of a sudden you need most of the processing power of your beast. Well, you can "move" instances to other servers. If we take this one step farther, you can actually have a virtualized infrastructure that would allow you to add hardware without changing anything. This type of setup can be cheaper, more flexible, and efficient than its traditional counterpart.

So we've covered how virtualization helps with servers in general. It can be a big help in supporting legacy HMI/SCADA technologies. It's really good for programs that are tough to configure (ah choo-Linux setups-oo). It seems less important for FactorySQL and FactoryPMI - they're already pretty good about being easy to install or move and having a lightweight footprint - especially on the client end with Java Web Start. You could set up a virtualized "production" and "testing" environment, both on the same computer, but this is pretty pointless since each installation would be better separate, and each could support the entire network on a desktop PC. I could see bigger setups greatly benefit from running virtualized instances.

Oh my - FDA 21CFR11

Read the reg!

I don't know whether to laugh or cry at Title 21 of the Food and Drug Administration's Code of Federal Regulation Part 11. It dictates requirements for Electronic Documents and Signatures. It seems to written on technical topics by non-technical people. It's well intended, but deliberately vague on implementation standards - over the top in many cases. The funny thing is that most vendors tout a "Certified 21CFR11 compliant" stamp on their hardware or software. The sad thing is that end users fail to realize that 80% or so of the requirements have to do with their: processes, training, documentation, etc.

I was getting so frustrated by a couple of users asking specifically how to implement a 21CFR11 application in FactoryPMI. They wanted point by point descriptions all the way down the list. I was first approached in 2005, but don't know of any users who have actually attempted a fully compliant project. I've had other users working in a 21CFR11 "compliant" facility that just wanted to use FactorySQL or FactoryPMI beside the application. I had to bite my tongue when one said that a hardware datalogger satisfied that requirement. Did anyone there actually read the regulation? Oh gosh! Maybe a good integrator helped them make their whole process compliant. I know for a fact that you can't throw any single piece of hardware or software and make it happen. Things like 11.300(e):

Initial and periodic testing of devices, such as tokens or cards, that bear or generate identification code or password information to ensure that they function properly and have not been altered in an unauthorized manner.

Carl made me feel better when he pointed out that 21CFR11 isn't a compliance standard for HMI/SCADA design software. Its a compliance standard for HMI/SCADA project implementations, which are two totally separate things. So, I wrote a white paper on creating a compliant 21CFR11 application with Inductive Automation software and didn't feel so bad about all the, "it is the responsibility of the designer..." answers. I might add that the "big vendors" white papers have similar responses.

I'd like to see someone actually do this - but it'd take planning on the front end and someone with experience. It would be absurd for an Integrator or end user to take this on from scratch, without help, as a first project.

Fascinating interview with Sun CEO

I was recently directed to read an interview with Sun Microsystem's CEO/COO, Jonathan Schwartz regarding his Open Source business strategy. This is particularly relevant given their new MySQL acquisition.

http://blogs.cnet.com/8301-13505_1-9757417-16.html

Jon stressed the necessity of driving adoption to generate revenue. He claims that seemingly competing products like Red Hat and Solaris complement each other because both are going for the same goal - look out M$! He makes a distinction between his "community" (users) and "market" (paying customers), insisting that one leads to the other. A casual user or developer shouldn't pay for his software, while that same individual might recommend purchasing it with a hefty support plan for a mission critical operation.

Sounds good to me! Establish a solid brand. Sell "systems" and "the experience" instead of hardware or software (think Apple). Provide an abundance of value without squeezing pennies from your customers - this also acts as free advertising. We'll see to what extent a certain industrial software company with killer products can follow suit.