I just finished participating in the OSIsoft vCampus Introductory Webinar. The OSIsoft vCampus is a new community designed to assist developers in developing applications around PI. This is a fantastic initiative in my opinion.

I have been working with PI for about 12 years. In the beginning there was the PI server, DataLink, and ProcessBook. It was pretty easy to sit down and get started. As the product mix grew I ran into the problem that it was not easy to get my hands on the new products and test them out. Once I became a consultant it was even worse because I was not a customer and there was no way to establish a formal relationship with OSIsoft.

I joined the OSIsoft Partner Program last year and was able for the first time as a consultant to establish a formal relationship with OSIsoft and get my hands on all of their products. In the past I had to wait until one of my customers purchased a product in order to get the opportunity to play with it and learn to use it. The Partner Program allowed me for the first time to test products and work on demo applications for everything that OSIsoft produced. This was good for me and I’m sure that it was ultimately good for them.

This year they are taking the concept one step further and introducing vCampus. The site offers a reference library containing all of the product documentation in one easy location. More importantly, they are adding white papers and field notes to help with common problems and to share best practices. The site offers discussion forums with direct OSIsoft developer participation and Blogs produced by developers and product managers.

The vCampus subscription provides me with a developer license for all of OSIsoft’s products. This is a tremendous asset as I am able to get up to speed on new technologies and experiment with new concepts. The site offers a DVD image download containing all of up to date products. New images will be released frequently. Ultimately a Microsoft Updates type of service is planned. The licensing is pretty robust and allows the developer to build up to a 3 server collective in order to test HA configurations and also test applications against HA.

Finally the vCampus allows developers access to the PI SDK and AF SDK. A couple of years ago OSIsoft announced that they would curtail access to the SDK’s. They later decided that they needed to provide access to the SDK’s in order to encourage development around PI but they also needed to maintain the quality of applications and prevent developers from creating poorly constructed programs that might cause problems.

It looks like OSIsoft has devoted 3 full time people to vCampus. This should help to insure that the initiative gets the traction that it needs to really take off. 

You can checkout vCampus and join at :http://vCampus.OSIsoft.com

Everyone in the CEMS business has been waiting to see how this goes and now we know. A federal court has struck down the EPA’s cap and trade regulation for Mercury.

The Associated Press: EPA’s Relaxed Emissions Rule Struck Down.

This means that the EPA will have to go back and revise their mercury regulations. It will be interesting to see how they handle this. For the time being I guess it is safest to assume that the time lines for installing and certifying you mercury equipment are unchanged.

Thermo recently released a press release saying that they had 25 systems that have passed all of the required regulatory testing (I’m sure that they mean as far as they can given the state of the regulations and standards at this point). Hopefully we will soon be able to add a couple more to the list.

Here is a link to the release:

Thermo Scientific Mercury Freedom System Demonstrating Complianc.

The installation of the MCG’s has been going a little rough. At this time we have one of the units installed but not completely tested.

We have the first production “X Box” systems and the design has been updated and standardized since we installed ours. As a result there are a few standard parts that didn’t fit in our box the way they were indented to so some field modifications had to be made. We also had the types of problems that you always seem to have when you have a system that has been operating for months. When you turn it off, it doesn’t always like to come back up. We had to replace one of the regulators.

The nitrogen generator has also been installed on that one unit. It took a little while to get the analyzer tuned back in after that. The PMT voltage had to be adjusted quite a bit. I’ll try to put together a few before and after plots to show how the data has been effected.

We did run into a problem with one of the systems just prior to the MCG install. It started to fail span cals. A couple of split nylon fittings were found. They were parts that were to be removed as part of the MCG install anyway.

We have also added some new thermocouple readings to the system. Our choice to install a PLC on the stack has started to pay dividends as it was relatively easy to add new signals to the system. We added a thermocouple to the stinger heater so that we can monitor its operation. We also added the second umbilical heater zone temperature.

We will have a RATA team in tomorrow to start performing EPA Method 30B tests to prove out the systems and satisfy the contract requirements. Thermo is recommending that you check the output of your calibrator during the RATA. The plan is to run a couple of traps off of the analyzer and have them analyzed to insure that the calibration gas generator is operating properly.

If things go reasonable well tomorrow I should have a chance to post about the RATA and the first few calibrations with the MCG installed.

We are working on the last few steps to complete the project.

Thermo sent us a mercuric choloride generator for a game cube probe but we have the modified inertial probe installed so we had to wait a couple of months for the right kit. We are scheduled to have it installed the week of the 14^th. If all goes well that will be the last piece of hardware to install.

We have tentatively scheduled a 30A RATA test for the last week of January. This will of course not be for official certification but GE is obligated to perform the test in order to prove out the system.

A long overdue update…

All of the little issues have finally been resolved and both Mercury Freedom Systems are working. We have the mercuric chloride generators and nitrogen generators on order and expect to receive them in the next few weeks.

We encountered a number of difficulties during the commissioning of the systems. Our original plan to commission the systems in two weeks turned into a two month project. To their credit, Thermo stuck to the job and eventually resolved all of the issues. We have been very happy with their dedication and tenacity in getting the systems to work. I think that it is pretty amazing that whenever we needed a new part they were able to ship it to us the next day. At this point in the game I would have expected shortages in parts but they have apparently done a very good job on the manufacturing end and it shows.

It is becoming apparent that everyone at Thermo has a lot on their plate and I expect that the plates will only be piled higher as more systems are installed. If you got in the game early then you were smart. If you are just getting started I would say that you will probably have to be patient. I remember many days in 1994 sitting on the phone trying to get support as we installed new CEMS. Everyone was overloaded with the impending 1/1/95 deadline and it sometimes took a long time just to get to the right person to fix the problem. I know that everyone is gearing up for the rush but there will be so many people installing systems that I think that we can realistically expect everyone to be a little more shorthanded than they would prefer. This is just the nature of the beast and in my opinion an avoidable consequence of the EPA not taking a more phased approach to implementing these systems. I think that it is very hard to run a business when you have to gear up to meet this huge rush knowing that things are likely to settle down again after the deadlines pass. That is what happened in ’95 after all of the CEMS were installed. Everyone hired like mad heading into the deadline and then turned around and laid off afterwards. That may not happen this time because overall the regulatory pressures are more intense and the systems are even more complex and will likely require more support. We’ll have to wait and see how it plays out.

I guess I should provide a summary of some of the problems that we encountered during the install. We ordered the first productions 82X (dubbed the “X Box”) systems. These systems take the probe controller components and place them on the stack. This design allows the heaters and thermocouples in the umbilical to terminate on the stack and not in the shelter which eliminates the likelihood that lightning will cause problems in the shelter. This is generally considered to be a good design decision in Florida which is the lightning capital of the US.

The approach that Thermo took with the X Box was to basically turn the umbilical upside down. The regular probe controller components were taken out of the blue rack mount box and placed in a NEMA 4X enclosure. All of the terminations that are regularly made in the shelter are made on the stack and the umbilical terminates at the probe with a short piece of umbilical running from the probe controller to the probe. The RS-485 signals from the analyzer in the shelter pass through a fiber converter and travel through fiber optic cable to another converter in the X Box NEMA enclosure which then plugs into one of the probe controller circuit boards as an RS-485 connection again.

I’ve heard several people make a big deal about the fiber like it is some exotic new technology. I’m not sure why this is as converting serial signals and Ethernet to fiber and back again is a no brainer these days. I do it all the time and never run into any problems. We had zero trouble with this aspect of the design except for the fact that some of the DIP switches on the converter were not set properly from the factory. A quick Google search for the product literature and a few switch changes and we were up and running.

The first problem that we identified with the X Box was the result of improper wiring. We originally planned two 30A 220V circuits to the cabinet to run the umbilical heaters and one 20A 120V circuit. This is what we were told would be required but when the box arrived they had modified the design so that it only required the 220V feeds. A converter in the box supplied the 120V where it was needed. This is a good thing except that it wasn’t wired correctly internally and because this was the first system out the door, the wiring diagrams had not been completed. It took some time to figure everything out and get it working. The I&C techs and GE field service guys were not really happy about this. The bad thing about the box being on the stack is that it is usually around 108 degrees up there in the summer. It really isn’t fun trying to troubleshoot electrical wiring while you are soaked in sweat.

We quickly ran into another problem on the stack. The solenoid valves on the probe were not operating properly. We did a lot of head scratching before we finally checked continuity on the wiring in the umbilical piece running between the probe controller and the probe. We discovered that one end of the wire wasn’t really connected to the other end. We ordered another short umbilical piece and it had the same problem. The third time was the charm. It is my understanding that Technical Heaters supplies these parts and they really shouldn’t have shipped these parts without QA’ing them better.

We also ran into some problems with the signals between the probe controller and the probe. I think that most of those were caused by grounding and shielding issues in the cabling. I got married in July and was wine tasting in Napa or hanging out around Monterey Bay around the time that they guys were on the 108 degree stack working this one out.

There is one aspect of the design of the probe controller that leaves a lot to be desired. The probe controller can control a maximum of two umbilical heater zones. Each umbilical heater zone can run 200-250ft so a number of users will have more than two zones. You aren’t going to be able to control them with the probe controller. You are going to have to go to a Watlow and wire it to your PLC for indication. You are better doing that even if you only have two zones as we do. Each heater zone has its own thermocouple but the probe controller has no place to connect the second thermocouple. It controls both zones from one thermocouple. I guess in theory this shouldn’t present a problem but it is the kind of thing that drive the guys in the field nuts and makes them question the intelligence of the guys in the lab. If you aren’t measuring your second zone, how would you know if it failed? I would think that a better design would have used off the shelf temperature controllers and provided plenty of 4-20mA inputs on the probe controller to provide indication. As it is I would recommend using Watlows and feeding the signal into your PLC or system controller and not use the Thermo probe controller for this function.

While we are talking about the issue of umbilical temperature I guess there are a few items to discuss. First off Thermo suspects that you really don’t need to keep the umbilical heated but they don’t have enough field data to prove it so they figure it is better to be safe than sorry. I think this is one of the reasons that they aren’t too concerned about monitoring that second heater zone. If it does go out, they suspect that it might not cause a problem. We don’t know that for sure. We are going to program the PLC to put the system in purge mode if we lose temperature indications just to be safe. At some point in time we will have to decide what we want to do long term. Luckily when we installed the NOx/SOx CEMS upgrade we put an Allen Bradley Micrologix 1100 DIN rail mounted PLC on the stack. This means that we have the capability of sending down the extra temperature signal. Having the Micrologix on the stack has given us the flexibility to respond to these types of issues and I am glad that we included it in the upgrades.

In addition to measuring the second umbilical zone we also chose to add a thermocouple to the stinger heater. This will also be connected to the Micrologix. Without this, there is no way to know if you stinger heater has failed. We also plan to add a probe enclosure temperature indication. During the many incursions into the probe during troubleshooting the temperature sensor in the converter wasn’t seated properly and we overheated the probe enough to begin to melt the Teflon tubing. This is bad business so we figure that adding that indication would have keyed us to the fact that we had a problem. Without it we thought that the converter heater wasn’t working properly when in fact it was cooking everything.

The system integration issues on the stack caused a number of problems which resulted in probe pluggage and other issues that all had to be fixed. This ate up extra time.

At this point the systems have been running for a while and the probes seem fine. We have a number of issues to address in the near future. The first item will be running a linearity. GE has had a number of questions to Thermo about how to do this and I don’t think that the systems have been acting entirely as expected. I believe that GE has this part figured out and we should be able to get it working around the beginning of November.

We are awaiting a firmware update from Thermo that will add the code to handle the Mercuric Chloride Generators. This has been delayed but we are hoping to get it in the next couple of weeks. Right now we are planning to install the MCG’s around Nov 12^th.

I said that I would be live blogging from the users conference. If you have ever been then you know how busy it can get. Things are pretty packed from morning until night.

We are heading into the end of the conference today and I guess I could say that I was just waiting to get a little perspective on things before I started shooting my mouth off.

A lot of things have changed since I was last at the conference in 2002. For the most part these changes are good and I’ll try to cover most of the high points.

I guess the most important thing to cover is the recent PR1 release of PI and the updated road map. The PR1 release is a major milestone and it really is the result of OSI realizing that the PI server needed some major new functionality to keep up with user needs.

A few years ago we (the users) started to identify some major roadblocks and impediments to growing larger PI systems and reliably collecting data. The two major problems were the buffering subsystem and the fact that there was no easy way to replicate PI data to more than one server. In effect you had all of your eggs in one basket and the way to get the eggs to the basket could be flakey at times.

The PR1 release introduced a new buffering subsystem with major improvements. The main thing that was added was a more transactional approach to submitting data to the PI server. In the past we had all had problems where an interface node would buffer data and then for one reason or other the buffer would be emptied but the data would not show up on the PI server. You had essentially lost the data with no way to recover it. The more transactional approach insures that the data makes it to the PI server. The buffer nodes checks to make sure that the data showed up in the archive before it deletes the information from the buffer.

The second major improvement is the High Availability (HA) PI “collective”. This upgrade to the PI server makes it possible for an interface node to feed several PI servers. It also makes it possible for multiple PI servers to synchronize their data. This is much improved from the PI to PI interface. Upgrades on the client end make this transparent to the users. They connect to PI as they always have but on the back end they may be connecting to one of many servers. This is HUGE. It allows you to take down a server for maintenance transparently. It means that if a server fails, your users never know. It also means that you could set up a server to service your users who hammer your server with huge historical data queries while another services your regular users. These are just a few examples but the architectural possibilities and your ability to engineer a systems that meets your availability, performance, and regulatory needs is greatly enhanced.

There are several other new features that are coming for the PR2 release. Some of them will be released in a bridge release prior to the PR2 release.

I’ll need to write a lot more about this stuff. I attended one of the PI server presentations presented by some of the OSI developers where they showed a 64bit Windows Server 2008 box that cost about $20,000 running with 21 million PI tags. 500,000 of those tags were being updated every second. The tags were separated into different partitions each with its own snapshot table and set of archives. This allowed them to support future data, a long time items on a lot of wish lists. It also means that the archive data is being written across multiple physical disk spindles which alleviates another performance bottleneck. While it is one thing to do this in the lab and another thing entirely to have a system in the field, it is good to know that they are working to stay ahead of demand.

There are a lot of other new things that I need to cover. The licensing is being updated with Enterprise agreements and site licenses which will free you from point count constraints. This is a big change that I need to cover. There are also a lot of new tools out there to cover.

 Please check back for additional post over the next few days.

Quite a bit of progress has been made. Unit 1 is up and running. Thermo was installing the lamp upgrade in the analyzer and the calibration gas hydrator today. Below is a picture of the hydrator.

It is mounted on the outside of the cabinet with a growing number of components. I will take another picture of the cabinet and post later. I’ll also let you know how it works.

Most of the problems that we were having stemmed from some wiring mistakes that were made when the “X box” cabinet was assembled. The “X box” is basically the components from the standard blue box probe controller that have been mounted in a NEMA enclosure suitable for the stack environment. Once the wiring problems were identified they were not that difficult to fix.

Unit 2 was still experiencing a few problems that are probably due to a leak in the probe tubing. Hopefully that won’t be too hard to fix.

Hopefully by the end of this week both systems will be running and have the hydrator and lamp upgrade installed. We will hopefully be receiving the nitrogen generators and the mercuric chloride generators soon.

I’ll post some more details as soon as I can.

I will be live blogging from the OSISoft Conference in a couple of weeks. It has been about 3 years since I have been able to make it to a conference and I am really looking forward to catching up on the latest developments.

We have been working on the Thermo Fisher Scientific Mercury Freedom installation for 3 weeks and have hit a series of snags. We are the first production installation (there is 1 R&D unit running) of the probe controller on the stack using the 82X configuration dubbed the “X Box”. Basically we did not want to run copper lines down the stack because here in Flordia we call that a lightning rod. We instead opted to install the probe controller on the stack and use fiber converters to connect the analyzer and the probe controller which normally sit in the rack together and use RS-485 over copper. The communication via fiber seems to work well and I must say that the analyzer and calibrator with their i-Series displays and menu system are very easy to use. The analyzer coordinates the actions of the calibrator and the probe controller. Thermo excels at this sort of thing and those components are impressive.

The 82X design basically takes the guts out of the standard Thermo blue box and puts them in a NEMA enslosure that is mounted on the stack. Fiber converters carry the RS-485 signals but other than that it is supposed to be identical to the regular probe controller.The long umbilical runs from the analyzer and calibrator up the stack and into the probe enclosure. The long umbilical going up the stack contains only tubing, thermocouples, and heat trace (the thermocouples and heat trace are terminated on the stack inside the X Box). Another short umbilical connects the probe controller box and the probe. This short umbilical contains tubing and a series of wires which allow the probe controller to read the thermocouples, power the heat traces, and control all of the probe components such as solenoid valves, the large ball valve, and heaters for the probe, stringer, and converter. This short umbilical has caused many of the problems. We discovered that the first set (1 for each unit we are installing) contained some bad wires meaning that there was no continuity from one end to the other. We were sent replacements which I am told have the same problem. For now we are using outside wires to carry these signals.

We have discovered some problems with the umbilical heater circuits. This has been a confusing issue from the beginning because Thermo states that you need a heater zone for each 200ft of umbilical. Our lines are in the 500+ ft range so I originally planned on 3 zones but the probe controller will only support one or two zones so we have 2 zones of about 250+ ft each. Right now we cannot get the probe controller to properly read the two zones and there are some problems with the control circuitry so we cannot maintain the proper temperature. We received upgraded controller boards since they have been tweaked since our unit shipped but this did not correct all of the problems. This could be a grounding or shielding issue and we have some work to do to get to the bottom of it. At this point we have shut the systems down with a purge to keep everything clean and we are working with Thermo to determine the next step.

It is expected to have some hiccups at this early stage. I am told that Thermo has about 100 systems shipped so far and I’m sure that most of those have not yet been installed. I’m pretty confident that we will get past these problems.

On another note, I received confirmation that your calibration gas hydrators, a firmware upgrade, and a lamp controller update for the analyzer were all shipped and should arrive next week.

I’ll post again next week with more details. I imagine that it will take a few weeks to get everything straight.