The time has come to renew my OSIsoft vCampus subscription. This is the second year of the vCampus program and my second year participating. In case you are not familiar with vCampus, you can find the site here http://vcampus.osisoft.com.

I’ve been working with PI since PI 3 was released on Windows NT around 1996. I have been earning a living primarily as a PI professional services provider since 2000. In the early days, I was only able to get my hands on new PI software when a customer ordered it. This was frustrating because I ran into the classic chicken and egg scenario where I could not try software and work on demos until a customer purchased it and customers were reluctant to purchase the new software because I had not tried it. OSIsoft addressed the program a few years ago with their partner program. Partners received licenses for all of the PI software. For the first time, I was able to build sand box environments and try everything.

Last year OSIsoft decided to improve on this model and start the virtual campus or vCampus site. As a developer I subscribe to the vCampus site for an annual fee of $1500. I receive a logon to the vCampus site and have access to pretty much everything I need to develop applications around the OSIsoft PI system. Through vCampus I have access to all of the PI software including many early beta releases. The vCampus site is a virtual community with hundreds of developers working on PI. We communicate on forums and exchange ideas and work out problems. OSIsoft developers post blog entries to give us a windows into what they are thinking. OSIsoft also hosts many webinars to explore topics suggested by the community. I also have access to all of the official documentation as well as many white papers. I would say that the vCampus site has greatly enhanced the development community that has grown around the PI products.

I also have a Microsoft Developers Network (MSDN) subscription. This gives me access to all of the Microsoft software as well as access to developer forums and other items. The combination of MSDN and vCampus allows me to keep up to date with all of the new releases as they come out. When customers decide to deploy something, it is likely that I have already been working with it for months.

I am configuring the following in a VMWare Workstation environment:

VM 1:

• Windows 2003 Domain Controller

VM 2:

• Windows 2008
• SQL Server 2008
• Microsoft Sharepoint Server Enterprise 2003 SP2 with Excel Services
• PI Server 3.4.380 (Windows Integrated Security)
• Application Framework (AF) Server 2.1
• PI Advanced Calculation Enginer (ACE) 2.1
• Rt Webparts (Sharepoint Portal)
• PI DataLink for Excel Services
• PI OLEDB Provider 3.3.0.1
• PI SQL Data Access Server (DAS) 32 bit
• PI to PI interface
• PI System Management Tools 3.3.1.3

Having this environment already configured in VMWare offers many advantages.

1. I can save a snapshot of the system prior to experimenting. This allows me to easily return to the starting point after the experiment.
2. I can easily deploy a copy of the environment and begin developing a prototype for a customer. I do not have to spent a lot of time configuring a development environment. The customer gets better bang for their buck.
3. I can work on my workstation at the office or copy the VM to my laptop and take it on the road.
4. While on the road I can use the PI to PI interface to feed live data from a production PI system to develop prototypes without the risk of interfering with that production environment. If available, I can also use a VPN connection to do the same thing. I can very quickly configure a prototype for the customer to try without ever having to set foot on site. In today’s world of reduced budgets and tight timelines, this is a perfect development scenario.

I have also done some experimenting with Terminal Services Gateway. This enables me to remotely run applications in my development environment. This could be very convenient when I am on the road as I can leave everything running on my workstation and just access it remotely via my wireless card. I could also easily give access to customers. Of course I was able to do this sort of stuff back in 1992 on Unix systems but it is still exciting to be able to do it with my favorite PI software today.

Interesting predictions about the growth in the CEMS market can be found here.

It is pretty likely that as the EPA imposes restrictions and taxes on CO2 emissions that the need to maintain reliable and accurate CEMS will become even more important.

It will also be interesting to see how the Hg CEMS market progresses. As the article mentions, many plants have Hg CEMS that have been moth balled. I have already seen instances where state agencies are starting to arm twist to get data from these systems. They may not be required by the EPA but the states definitely want this data and they are pressuring utilities to provide it. I predict new regulations and even where new regulations are not enacted I suspect that states will start negotiating to add this data to reporting requirements when operating permits are renewed.

I just finished participating in the OSIsoft vCampus Introductory Webinar. The OSIsoft vCampus is a new community designed to assist developers in developing applications around PI. This is a fantastic initiative in my opinion.

I have been working with PI for about 12 years. In the beginning there was the PI server, DataLink, and ProcessBook. It was pretty easy to sit down and get started. As the product mix grew I ran into the problem that it was not easy to get my hands on the new products and test them out. Once I became a consultant it was even worse because I was not a customer and there was no way to establish a formal relationship with OSIsoft.

I joined the OSIsoft Partner Program last year and was able for the first time as a consultant to establish a formal relationship with OSIsoft and get my hands on all of their products. In the past I had to wait until one of my customers purchased a product in order to get the opportunity to play with it and learn to use it. The Partner Program allowed me for the first time to test products and work on demo applications for everything that OSIsoft produced. This was good for me and I’m sure that it was ultimately good for them.

This year they are taking the concept one step further and introducing vCampus. The site offers a reference library containing all of the product documentation in one easy location. More importantly, they are adding white papers and field notes to help with common problems and to share best practices. The site offers discussion forums with direct OSIsoft developer participation and Blogs produced by developers and product managers.

The vCampus subscription provides me with a developer license for all of OSIsoft’s products. This is a tremendous asset as I am able to get up to speed on new technologies and experiment with new concepts. The site offers a DVD image download containing all of up to date products. New images will be released frequently. Ultimately a Microsoft Updates type of service is planned. The licensing is pretty robust and allows the developer to build up to a 3 server collective in order to test HA configurations and also test applications against HA.

Finally the vCampus allows developers access to the PI SDK and AF SDK. A couple of years ago OSIsoft announced that they would curtail access to the SDK’s. They later decided that they needed to provide access to the SDK’s in order to encourage development around PI but they also needed to maintain the quality of applications and prevent developers from creating poorly constructed programs that might cause problems.

It looks like OSIsoft has devoted 3 full time people to vCampus. This should help to insure that the initiative gets the traction that it needs to really take off. 

You can checkout vCampus and join at :http://vCampus.OSIsoft.com

Everyone in the CEMS business has been waiting to see how this goes and now we know. A federal court has struck down the EPA’s cap and trade regulation for Mercury.

The Associated Press: EPA’s Relaxed Emissions Rule Struck Down.

This means that the EPA will have to go back and revise their mercury regulations. It will be interesting to see how they handle this. For the time being I guess it is safest to assume that the time lines for installing and certifying you mercury equipment are unchanged.

Thermo recently released a press release saying that they had 25 systems that have passed all of the required regulatory testing (I’m sure that they mean as far as they can given the state of the regulations and standards at this point). Hopefully we will soon be able to add a couple more to the list.

Here is a link to the release:

Thermo Scientific Mercury Freedom System Demonstrating Complianc.

The installation of the MCG’s has been going a little rough. At this time we have one of the units installed but not completely tested.

We have the first production “X Box” systems and the design has been updated and standardized since we installed ours. As a result there are a few standard parts that didn’t fit in our box the way they were indented to so some field modifications had to be made. We also had the types of problems that you always seem to have when you have a system that has been operating for months. When you turn it off, it doesn’t always like to come back up. We had to replace one of the regulators.

The nitrogen generator has also been installed on that one unit. It took a little while to get the analyzer tuned back in after that. The PMT voltage had to be adjusted quite a bit. I’ll try to put together a few before and after plots to show how the data has been effected.

We did run into a problem with one of the systems just prior to the MCG install. It started to fail span cals. A couple of split nylon fittings were found. They were parts that were to be removed as part of the MCG install anyway.

We have also added some new thermocouple readings to the system. Our choice to install a PLC on the stack has started to pay dividends as it was relatively easy to add new signals to the system. We added a thermocouple to the stinger heater so that we can monitor its operation. We also added the second umbilical heater zone temperature.

We will have a RATA team in tomorrow to start performing EPA Method 30B tests to prove out the systems and satisfy the contract requirements. Thermo is recommending that you check the output of your calibrator during the RATA. The plan is to run a couple of traps off of the analyzer and have them analyzed to insure that the calibration gas generator is operating properly.

If things go reasonable well tomorrow I should have a chance to post about the RATA and the first few calibrations with the MCG installed.

We are working on the last few steps to complete the project.

Thermo sent us a mercuric choloride generator for a game cube probe but we have the modified inertial probe installed so we had to wait a couple of months for the right kit. We are scheduled to have it installed the week of the 14^th. If all goes well that will be the last piece of hardware to install.

We have tentatively scheduled a 30A RATA test for the last week of January. This will of course not be for official certification but GE is obligated to perform the test in order to prove out the system.

A long overdue update…

All of the little issues have finally been resolved and both Mercury Freedom Systems are working. We have the mercuric chloride generators and nitrogen generators on order and expect to receive them in the next few weeks.

We encountered a number of difficulties during the commissioning of the systems. Our original plan to commission the systems in two weeks turned into a two month project. To their credit, Thermo stuck to the job and eventually resolved all of the issues. We have been very happy with their dedication and tenacity in getting the systems to work. I think that it is pretty amazing that whenever we needed a new part they were able to ship it to us the next day. At this point in the game I would have expected shortages in parts but they have apparently done a very good job on the manufacturing end and it shows.

It is becoming apparent that everyone at Thermo has a lot on their plate and I expect that the plates will only be piled higher as more systems are installed. If you got in the game early then you were smart. If you are just getting started I would say that you will probably have to be patient. I remember many days in 1994 sitting on the phone trying to get support as we installed new CEMS. Everyone was overloaded with the impending 1/1/95 deadline and it sometimes took a long time just to get to the right person to fix the problem. I know that everyone is gearing up for the rush but there will be so many people installing systems that I think that we can realistically expect everyone to be a little more shorthanded than they would prefer. This is just the nature of the beast and in my opinion an avoidable consequence of the EPA not taking a more phased approach to implementing these systems. I think that it is very hard to run a business when you have to gear up to meet this huge rush knowing that things are likely to settle down again after the deadlines pass. That is what happened in ’95 after all of the CEMS were installed. Everyone hired like mad heading into the deadline and then turned around and laid off afterwards. That may not happen this time because overall the regulatory pressures are more intense and the systems are even more complex and will likely require more support. We’ll have to wait and see how it plays out.

I guess I should provide a summary of some of the problems that we encountered during the install. We ordered the first productions 82X (dubbed the “X Box”) systems. These systems take the probe controller components and place them on the stack. This design allows the heaters and thermocouples in the umbilical to terminate on the stack and not in the shelter which eliminates the likelihood that lightning will cause problems in the shelter. This is generally considered to be a good design decision in Florida which is the lightning capital of the US.

The approach that Thermo took with the X Box was to basically turn the umbilical upside down. The regular probe controller components were taken out of the blue rack mount box and placed in a NEMA 4X enclosure. All of the terminations that are regularly made in the shelter are made on the stack and the umbilical terminates at the probe with a short piece of umbilical running from the probe controller to the probe. The RS-485 signals from the analyzer in the shelter pass through a fiber converter and travel through fiber optic cable to another converter in the X Box NEMA enclosure which then plugs into one of the probe controller circuit boards as an RS-485 connection again.

I’ve heard several people make a big deal about the fiber like it is some exotic new technology. I’m not sure why this is as converting serial signals and Ethernet to fiber and back again is a no brainer these days. I do it all the time and never run into any problems. We had zero trouble with this aspect of the design except for the fact that some of the DIP switches on the converter were not set properly from the factory. A quick Google search for the product literature and a few switch changes and we were up and running.

The first problem that we identified with the X Box was the result of improper wiring. We originally planned two 30A 220V circuits to the cabinet to run the umbilical heaters and one 20A 120V circuit. This is what we were told would be required but when the box arrived they had modified the design so that it only required the 220V feeds. A converter in the box supplied the 120V where it was needed. This is a good thing except that it wasn’t wired correctly internally and because this was the first system out the door, the wiring diagrams had not been completed. It took some time to figure everything out and get it working. The I&C techs and GE field service guys were not really happy about this. The bad thing about the box being on the stack is that it is usually around 108 degrees up there in the summer. It really isn’t fun trying to troubleshoot electrical wiring while you are soaked in sweat.

We quickly ran into another problem on the stack. The solenoid valves on the probe were not operating properly. We did a lot of head scratching before we finally checked continuity on the wiring in the umbilical piece running between the probe controller and the probe. We discovered that one end of the wire wasn’t really connected to the other end. We ordered another short umbilical piece and it had the same problem. The third time was the charm. It is my understanding that Technical Heaters supplies these parts and they really shouldn’t have shipped these parts without QA’ing them better.

We also ran into some problems with the signals between the probe controller and the probe. I think that most of those were caused by grounding and shielding issues in the cabling. I got married in July and was wine tasting in Napa or hanging out around Monterey Bay around the time that they guys were on the 108 degree stack working this one out.

There is one aspect of the design of the probe controller that leaves a lot to be desired. The probe controller can control a maximum of two umbilical heater zones. Each umbilical heater zone can run 200-250ft so a number of users will have more than two zones. You aren’t going to be able to control them with the probe controller. You are going to have to go to a Watlow and wire it to your PLC for indication. You are better doing that even if you only have two zones as we do. Each heater zone has its own thermocouple but the probe controller has no place to connect the second thermocouple. It controls both zones from one thermocouple. I guess in theory this shouldn’t present a problem but it is the kind of thing that drive the guys in the field nuts and makes them question the intelligence of the guys in the lab. If you aren’t measuring your second zone, how would you know if it failed? I would think that a better design would have used off the shelf temperature controllers and provided plenty of 4-20mA inputs on the probe controller to provide indication. As it is I would recommend using Watlows and feeding the signal into your PLC or system controller and not use the Thermo probe controller for this function.

While we are talking about the issue of umbilical temperature I guess there are a few items to discuss. First off Thermo suspects that you really don’t need to keep the umbilical heated but they don’t have enough field data to prove it so they figure it is better to be safe than sorry. I think this is one of the reasons that they aren’t too concerned about monitoring that second heater zone. If it does go out, they suspect that it might not cause a problem. We don’t know that for sure. We are going to program the PLC to put the system in purge mode if we lose temperature indications just to be safe. At some point in time we will have to decide what we want to do long term. Luckily when we installed the NOx/SOx CEMS upgrade we put an Allen Bradley Micrologix 1100 DIN rail mounted PLC on the stack. This means that we have the capability of sending down the extra temperature signal. Having the Micrologix on the stack has given us the flexibility to respond to these types of issues and I am glad that we included it in the upgrades.

In addition to measuring the second umbilical zone we also chose to add a thermocouple to the stinger heater. This will also be connected to the Micrologix. Without this, there is no way to know if you stinger heater has failed. We also plan to add a probe enclosure temperature indication. During the many incursions into the probe during troubleshooting the temperature sensor in the converter wasn’t seated properly and we overheated the probe enough to begin to melt the Teflon tubing. This is bad business so we figure that adding that indication would have keyed us to the fact that we had a problem. Without it we thought that the converter heater wasn’t working properly when in fact it was cooking everything.

The system integration issues on the stack caused a number of problems which resulted in probe pluggage and other issues that all had to be fixed. This ate up extra time.

At this point the systems have been running for a while and the probes seem fine. We have a number of issues to address in the near future. The first item will be running a linearity. GE has had a number of questions to Thermo about how to do this and I don’t think that the systems have been acting entirely as expected. I believe that GE has this part figured out and we should be able to get it working around the beginning of November.

We are awaiting a firmware update from Thermo that will add the code to handle the Mercuric Chloride Generators. This has been delayed but we are hoping to get it in the next couple of weeks. Right now we are planning to install the MCG’s around Nov 12^th.

I said that I would be live blogging from the users conference. If you have ever been then you know how busy it can get. Things are pretty packed from morning until night.

We are heading into the end of the conference today and I guess I could say that I was just waiting to get a little perspective on things before I started shooting my mouth off.

A lot of things have changed since I was last at the conference in 2002. For the most part these changes are good and I’ll try to cover most of the high points.

I guess the most important thing to cover is the recent PR1 release of PI and the updated road map. The PR1 release is a major milestone and it really is the result of OSI realizing that the PI server needed some major new functionality to keep up with user needs.

A few years ago we (the users) started to identify some major roadblocks and impediments to growing larger PI systems and reliably collecting data. The two major problems were the buffering subsystem and the fact that there was no easy way to replicate PI data to more than one server. In effect you had all of your eggs in one basket and the way to get the eggs to the basket could be flakey at times.

The PR1 release introduced a new buffering subsystem with major improvements. The main thing that was added was a more transactional approach to submitting data to the PI server. In the past we had all had problems where an interface node would buffer data and then for one reason or other the buffer would be emptied but the data would not show up on the PI server. You had essentially lost the data with no way to recover it. The more transactional approach insures that the data makes it to the PI server. The buffer nodes checks to make sure that the data showed up in the archive before it deletes the information from the buffer.

The second major improvement is the High Availability (HA) PI “collective”. This upgrade to the PI server makes it possible for an interface node to feed several PI servers. It also makes it possible for multiple PI servers to synchronize their data. This is much improved from the PI to PI interface. Upgrades on the client end make this transparent to the users. They connect to PI as they always have but on the back end they may be connecting to one of many servers. This is HUGE. It allows you to take down a server for maintenance transparently. It means that if a server fails, your users never know. It also means that you could set up a server to service your users who hammer your server with huge historical data queries while another services your regular users. These are just a few examples but the architectural possibilities and your ability to engineer a systems that meets your availability, performance, and regulatory needs is greatly enhanced.

There are several other new features that are coming for the PR2 release. Some of them will be released in a bridge release prior to the PR2 release.

I’ll need to write a lot more about this stuff. I attended one of the PI server presentations presented by some of the OSI developers where they showed a 64bit Windows Server 2008 box that cost about $20,000 running with 21 million PI tags. 500,000 of those tags were being updated every second. The tags were separated into different partitions each with its own snapshot table and set of archives. This allowed them to support future data, a long time items on a lot of wish lists. It also means that the archive data is being written across multiple physical disk spindles which alleviates another performance bottleneck. While it is one thing to do this in the lab and another thing entirely to have a system in the field, it is good to know that they are working to stay ahead of demand.

There are a lot of other new things that I need to cover. The licensing is being updated with Enterprise agreements and site licenses which will free you from point count constraints. This is a big change that I need to cover. There are also a lot of new tools out there to cover.

 Please check back for additional post over the next few days.

Quite a bit of progress has been made. Unit 1 is up and running. Thermo was installing the lamp upgrade in the analyzer and the calibration gas hydrator today. Below is a picture of the hydrator.

It is mounted on the outside of the cabinet with a growing number of components. I will take another picture of the cabinet and post later. I’ll also let you know how it works.

Most of the problems that we were having stemmed from some wiring mistakes that were made when the “X box” cabinet was assembled. The “X box” is basically the components from the standard blue box probe controller that have been mounted in a NEMA enclosure suitable for the stack environment. Once the wiring problems were identified they were not that difficult to fix.

Unit 2 was still experiencing a few problems that are probably due to a leak in the probe tubing. Hopefully that won’t be too hard to fix.

Hopefully by the end of this week both systems will be running and have the hydrator and lamp upgrade installed. We will hopefully be receiving the nitrogen generators and the mercuric chloride generators soon.

I’ll post some more details as soon as I can.