station goes offline, and is revived (again)

On November 3rd, 2009 I sent out an email reporting that the station had stopped transmitting again:

The PVGF1 Port Everglades station stopped transmitting this past weekend.  Its last transmission was sent at approximately 6:24pm local time on Saturday, October 31st.

I know you [Jack Stamates] were planning to visit the station after the cruise is over.  You might want to involve myself or Mike Shoemaker in the trip, or possibly, one of us Mikes might want to visit the station earlier than that, if we have the ability to open the box and permission to visit the Naval Base.

I'll look to see if there are any suspicious signs in the final transmissions but it looks like it's a clean break (i.e., no gradual power loss, no partial transmissions).  This may be a repeat of what happened last July.  Shoe fixed things that time, I think he may have just opened up the box and power-cycled the system.

Jack responded on November 5th, 2009 with the following update:

I pulled the power plug on the Campbell and the Sat transmiter, left the power off for a 30 sec then re inserted the plug...  Apparently this did not fix the problem.

Jack was able to visit the station again on November 16th, and on November 17th, 2009 I sent out the following update:

Jack visited the Port Everglades station again yesterday, and power-cycled the logger and transmitter.  He also swapped out the memory cards and brought me the retrieved card.  This didn't have any effect on transmissions (i.e., we are still offline, and have been since October 31st).  But I now have a memory card to answer some questions about the equipment.

First off, the datalogger seems to be perfectly ok.  It has been logging data nonstop since October 31st, right up until the card's retrieval yesterday.

Secondly, the transmitter is on, it is communicating with the logger, and it is continuing to acquire GPS fixes normally.  However, the error codes indicate that its failsafe mode has been tripped.  Basically it must have sensed conditions (voltage spike? temperature extreme?) that could have led to permanent damage, and it shut down its transmission systems.  Power-cycling will NOT reset the failsafe.  Instead, the "reset" button must be held down for at least 4 seconds, which will trigger a 60-second reset procedure.  It is entirely possible that this is all that is needed to get the station transmitting again.

Other considerations:  I currently have one other SAT-HDR-GOES transmitter, which I am testing on the roof.  It is not yet working.  I am also testing an updated pvgf1 logger program on the roof (among other things, this would add one of our old GE/Druck barometers, so I could sanity-check the air pressures returned by the WXT).  The new logger program (unlike the spare transmitter) IS working.

The bottom line is that our next logical step should probably be to do the transmitter-reset procedure on Jack's next visit.  I'd also like to visit the station to upgrade the programming and equipment, but I'm not ready yet.

Jack returned to Port Everglades and this time did a failsafe-reset on the satellite transmitter, which restarted the satellite transmissions.  I sent out the following update on November 19th, 2009:

Jack did a reset of the sat-hdr-goes transmitter at Port Everglades (pvgf1), and that seems to have done the trick.  We've received our first transmission from that station, with hopefully many more to come.  Good work, Jack!

low salinity readings, reliability of CT cleanings

On September 14th, 2009, Jack Stamates (FACE project) sent out an update about the Port Everglades CT.  He had noticed that the salinity readings had been unreasonably low, and in response he had contacted the dive company he was paying to do monthly instrument cleanings.

This resulted in an immediately resolution of the problem, and the CT salinity readings were once again reasonable.  It appears as though the contracted divers had not been adhering to the monthly CT cleaning schedule they'd agreed to.  Jack raised to issue to the awareness of the AOML dive collective and resolved to monitor the situation more closely from here on out.

regarding the accuracy of the station's barometer

[The following are excerpts of an email conversation between myself and Rex Hervey of NDBC.  When CHAMP first started feeding data from this station to NDBC, some questions were raised about the accuracy of its barometric pressure readings.  It so happened we had some similar questions about the WXT barometric pressures at LCIY2 at the same time.  In due course of time we determined that the pressure readings were all correct within instrument tolerances, and later (March 2010) we would add a redundant analog barometer to PVGF1 just to be sure.  Still, the model analyses shared by Rex are interesting enough to include on this blog.]

On June 25th, 2009, Rex wrote:

We are releasing the data from Port Everglades with the exception of pressure.  It looks like it is about one mb low.  We will continue to watch it.  If you can apply an offset, I can give you one after we have more time to look at it.  Picture is not up yet, but here is the web page: http://www.ndbc.noaa.gov/station_page.php?station=PVGF1

My reply on July 5th, 2009 was:

Can you tell me more about how you know the station's air pressure is off?  I assume you're comparing it with reports from surrounding stations but I'd like to know which ones.  The WXT520 weather transmitter is newly-calibrated and I'm not aware that we've had any problems with them in the past.

I could also put a standalone barometer out there for comparison purposes, although that will have to wait until we can coordinate another visit.

Rex followed up on July 6th, 2009 with:

It is compared to surrounding stations and numerical model analyses. Now that we have had a week or so to look at it, the bias is only about .3 to .4 mb low compared to model analyses.  That is within limits, so we will release it.

On August 12th I asked for another look at the situation, and Rex replied the same day with the following text and images:

I sent some plots comparing the baro reading with the NWS numerical model analyses.  Green Xs are from the model. One is over the last 2 months; the other is over the last month.  It appears as though it may be drifting closer to the model results over time.  Overall it is -.26 mb off, and more recently only -.15 mb off.  My initial evaluation after start up was only over a few observations.  I clearly should have given it more time.

One month.

Two months.

station goes offline, and is revived

On July 20th, 2009, I received an email from Rex Hervey (NDBC) about issues relating to data feeds (current and planned) from our St. Croix and Little Cayman CREWS stations.  As an afterthought Rex said:  "Also, we haven't received data from the Port Everglades station for a while now."

He wasn't the only one who'd noticed.  Tom Carsey on July 21st, 2009, sent his own trouble report:  "Evidently the Pt. Everglades site has not transmitted since July 12."

Following up on Rex's report (I was in Little Cayman helping install the new CREWS station at the time, and hadn't see Tom's message yet), I commented in email:

Rex at NDBC mentioned that our Port Everglades data feed went offline.  Checking the archives, it seems like it's been offline for about nine days.  It's difficult for me to investigate further from the field, but I thought I'd mention in case others wanted to look into it further.

The last transmission (that I see) was at 13:42 UTC on day 193 (Sunday morning, the 12th??).

Lew Gramer, who was the acting CHAMP sysadmin in my absence, followed up on July 21st, 2009:

Yes, the last transmission from the Port Everglades station that I see in the archives on our server is still the one on Sunday 12 July at 13:24 GMT. I checked the "usual" diagnostic data from the last day's transmissions (station and datalogger voltages, transmitter forward vs. reflected power, data and error counts, etc.), all seemed normal. Jack is out of town all week as well, but Shoe located that key and will travel to check the site out tomorrow... 

As Lew indicated, Mike Shoemaker (a/k/a "Shoe") traveled up to the Port Everglades site on the morning of July 22nd and was able to bring the station back online.  Lew sent out the following report about Mike's intervention later that afternoon:

Mike Shoemaker visited the Port Everglades monitoring station this morning, reset everything, opened the station door for a while and let it air out / cool off. The station has begun transmitting again: the 14:24 and 15:24 UT transmissions have been received, and were both well-formated. (Shoe, the last hourly transmission has reasonable looking salinity and sea temperature also.)

Shoe believes part of the reason for the sudden failure in early July may be that the enclosure holds too much heat. We checked, and the hourly mean "panel temperature" climbed above 36oC (97oF) or 50 of the hours when we got regular transmissions in late June and early July. Max PTemp was 38.8oC. (Shoe, went down from 32.7 last hour to 31.9 just now, so at least isn't getting worse.) Jack and Mike J., when you both get back, Shoe has ideas for how to fix this.

 Following Jack's next data-recovery visit on July 31st, I reviewed the patched data archives and was able to produce the following email summary of the incident on August 3rd, 2009:

There was NO interruption in datalogger activity in this period.  As far as I can tell from a brief look at the logger/transmitter diagnostics, the problem was specific to the transmitter.  Either the transmitter stopped working, or it merely stopped communicating with the datalogger.  Either way, Shoe's power-reset appears to have fixed the problem, although there's no guarantee that it won't happen again.

depths of the underwater instruments

 [The following is an excerpt from an email by Jack Stamates (FACE project) relating to the positions of the underwater sensors at this site.]

 On June 11th, 2009, Jack wrote:

The average depth of the ADCP is 1.9m and the CT sensor is 20-30cm higher in the water column than that.  So, lets say 1.65m.

The depth of the deep CT will depend on the topography where we put the mount but it will be about 12m-13m.

Since the rains have started there is definitely more flow out then before.

ADCP firmware, mounting stabilized, CT cleaning schedule

[The following is an excerpt from an email from Jack Stamates (FACE project) that touches on some topics concerned this station's ADCP, its underwater mounting, and plans for regular cleanings of the CT.]

On May 22nd, 2009, Jack writes:

To fill in some details, RD has offered us a firmware Beta version but I am reluctant to deploy it because, if it hangs the instrument up, I am in deep trouble.  (I would need to recover the instrument and open the case.)  Once the Firmware is declared operational (which should be soon) I will give it a try.  Today, the divers stabilized the mounting which was drifting a bit.  They also cleaned the CT sensor.  This will now be done monthly.  The divers reported that there was a fair amount of fouling.

programming update, station photos

A view of the completed PVGF1 station, showing the meterological sensors (foreground) and the USCG channel marker where the oceanographic sensors are mounted (background).  The station electronics are wholly contained within the box to the left.

The CREWS/FACE station at Port Everglades today had its datalogger programming updated, and at the same time its locally-stored data were collected from memory.  These data downloads will be used to "patch" the archives where any of the hourly satellite transmissions failed to get through, and they also include data from the ADCP which does not report its data to the datalogger for near-real time dissemination.

New programming updates will now make it possible for us to feed this station's data to the National Data Buoy Center (NDBC), whence they will be included in the WMO's Global Telecommunications System (GTS), making them available for use by national weather services all over the world.  Look for the station's NDBC reports at:  http://www.ndbc.noaa.gov/station_page.php?station=PVGF1

new station installed, a CREWS/FACE hybrid

A new monitoring station has been installed in the shipping channel leading to Port Everglades.  It is a collaboration between the Florida Area Coastal Environment (FACE) program and the Coral Reef Early Warning System (CREWS) technology developed by the Coral Health and Monitoring Program (CHAMP).  Both programs operate out of NOAA's Atlantic Oceanographic and Meteorological Laboratory (AOML).

Showing an overview of all operations:  the dive boat, the channel marker, the running of cables, the trench that was dug, the hard plastic tube that would be buried in the trench, and the sandbags that would be used to weight down the cables at the channel's bottom.

The electronics technicians at work.

The weatherproof box with the station's datalogger, satellite transmitter, and extensive battery backups.

The station includes meteorological sensors (wind, air pressure, air temperature, relative humidity, precipitation) and controlling electronics in a package installed on land, on property controlled by the US Navy.  These electronics are powered by a land-based power feed (essentially everything is just plugged into an electrical outlet) but incorporate a series of four rechargeable batteries capable of running the systems for more than a week even in the absence of the external power feed.  The land-based portion of the station is connected by cabling to two underwater instruments installed directly on the nearby USCG channel marker #7, an Acoustic Doppler Current Profiler (ADCP) and a Conductivity/Temperature sensor (CT).  Salinity measurements are calculated from the conductivities and temperatures reported by the CT.

All instruments save the ADCP will report in near-real time.  The ADCP will log its measurements internally and its data will be downloaded manually on a roughly monthly basis.

Current data are available online (http://www.coral.noaa.gov/static/data_pvgf1_Web_12.html) and will soon be fed automatically to NOAA's National Data Buoy Center (NDBC).

Port Everglades Station Online

New station deployed March 19, 2009, at

26.093 ° latitude, -80.109 ° longitude