station back online

[The following is a slightly edited version of an email I sent to Jim Hendee (CHAMP principal investigator) on January 13th, 2011.  It summarizes the results of a visit by Jack Stamates (FACE project) and myself to the station that morning, where our intention was to investigate what might have caused the station to stop transmitting on December 29th, 2010.]

On January 13th, 2011, I wrote:

Jack and I visited the station this morning.  The entire base (?) lost power for a week over new year's, Dec 27th to Jan 3rd.  [This was apparently due to a disastrous FP&L screwup.  Jack knows/understands more of the details of this.]

The station batteries performed beautifully and everything continued to operate normally (ADCP, CT, logger, WXT) except the sat transmitter.  So we lost no data whatsoever.  Jack was thrilled with this test of the station's operation during an extended power loss.  Great design!

The one weak point continues to be the ancient satellite transmitter.  At the battery levels recorded by the logger, even the transmitter should have continued to operate normally.  But something about those slightly-lower voltage levels, or possibly there were repeated voltage drops or spikes, something spooked the transmitter.  It was NOT in failsafe mode (I could see that when I connected directly to the transmitter) but it had stopped communicating with the logger, so it had no data to transmit.  Pushing the failsafe-reset button seems to have brought it back to life.  Remember, this is one of those old SAT-HDR-GOES transmitters, the last one (of five original) that still works at all.

Mike J+

Note that while Jack and I were visiting, we again downloaded all available ADCP data and swapped datalogger memory cards, an operation that normally occurs about once a month.

transmitter failure

[The following post is a copy of an email message I wrote to Jack Stamates on January 5th, 2011.  Transmissions from the Port Everglades station had stopped on December 29th, 2010, and we were planning to visit the site to see what may have gone wrong.]

On January 5th, 2011, I wrote:

I've updated all of my spreadsheets and data files with the most recently retrieved data card, from November 9th.  I've also had a closer look at the data from immediately before the station stopped transmitting, and there's a suggestion that something weird started happening a few days before.

The first hint of trouble was a skipped transmission on Monday, December 27th at 1pm local time.  When transmissions resumed in the next hour, the voltage diagnostics looked different than before.  The logger reports min, max and average voltage numbers -- normally the min numbers hang around 12.8V, the average at 13V, and the max around 13.8V.

Beginning Monday afternoon, the max and average value both dropped to about 12.8V and the min fell below 12.5V.  Things went on like that for about two days until transmissions ceased on Wednesday night (Dec 29th) at about 8pm local.

So this probably isn't the same transmitter-failure problem we've seen in the past.  The sure sign of that problem is that the GPS acquisition times go through the roof in the times leading up to the failure, and there was no sign of that in this case.

It would seem the station has a power issue of some kind.  This could mean that something weird is going on with the external power source; or the datalogger might be suffering some kind of power problem internally; or something connected to the logger might have short-circuited.  [In the past, we've seen something similar when an underwater instrument has a bulkhead failure and floods, shorting power to ground and draining the power supply of all the other instruments.]  There is no sign of trouble with the CT numbers that I can see in those past few days, or the meteorological instruments.

So anyhow, I'd suggest you prepare yourself for a wider range of problems when you go.  If you want me to meet you there, let me know, it's usually no trouble since I live so close.  Also, there's no guarantee that this is a transmitter-only problem, so you may find that the logger hasn't been storing the CT data to its memory card in the last week.

I've formatted your spare memory card so you can swap them out during your next visit.  I'll bring in by your office today.

Mike J+

data downloads, 2010

Between the station's equipment/programming update on April 16th, 2010 and its next on-site intervention on January 13th, 2011, FACE continued their routine of (roughly) monthly visits for downloading the ADCP's data reserves and swapping datalogger memory cards.  These visits gave us semi-regular access to ADCP data, which weren't included in the station's near-real time transmissions, allowed us to "patch" holes in our archives where occasional transmissions had been dropped, and gave us access to the more time-granular (5-second, 30-second, 1-minute, 6-minute) stored data that weren't included in the hourly transmission.

Between April 16th, 2010 and January 13th, 2011, these data-download visits took place on:

• May 19th, 2010
• June 24th, 2010
• August 6th, 2010
• September 10th, 2010
• November 9th, 2010

station goes offline, and is revived (yet again)

On August 16th, 2010, Jim Hendee (CHAMP principal investigator) asked about our automated status reports, that were showing an absence of data from Port Everglades.  I first replied briefly:

Port Everglades transmissions went offline on August 4th and have not resumed.  Probably a transmitter/GPS problem, but I'm working with the data downloads now and I'll send an update if/when I learn more. 

And then I followed up with a more detailed report:

I've poked into the data a bit more, so here's a fuller picture of what's going on with the Port Everglades station.

First of all, it's offline.  Its last transmission was at (local) noon on Wednesday, August 4th.  We know it's a transmitter-only problem because Jack (by chance) visited the station on Friday, August 6th and collected the latest memory card, so we can look at two more days' worth of data since the failure.

And it's worth saying that, as far as I know, FACE doesn't need satellite transmissions for its work.  I believe Jack uses the 6-minute granular CT data that I extract from the memory cards, not the near-real time transmissions.  Of course it's good to have the near-real time stuff so we know immediately if the equipment fails, but it's not (as far as I know) a must-have for FACE.

On the ICON side, we prefer to have the satellite transmissions working, because that's kinda what we do.  We track our uptime statistics and we feed our numbers to NDBC, so we're generally happier if the transmitter is working.

Anyhow, PVGF1 uses one of our old SAT-HDR-GOES transmitters, and it's been in steady decline since the end of April.  It seems like it's a problem with the GPS subsystem.  The old SAT-HDR-GOES can't transmit unless it gets a GPS fix before every transmission (by contrast the new TX312 will continue to transmit for one month even if all of the GPS satellites were to simultaneously fall into the ocean).  In April we started seeing GPS timeouts when it reached started to go 5 minutes without a fix (normally it takes less than a minute).

By the end of May, these timeouts were happening regularly, about once a day.  The odd thing is that a regular pattern developed where transmissions would fail for three or four hours every day beginning at about 1 PM local.  If I had to guess, I might say that the problem is exacerbated by higher temperatures in the box?

On August 4th the error codes switched from a GPS error to a failsafe error.  This means that the transmitter has probably kicked into failsafe mode again.  I think Jack knows how to reset the failsafe on the transmitter, and he should try doing this the next time he visits.

But the big picture is that the transmitter itself is probably failing.  One of the old SAT-HDR-GOES that we had at Molasses Reef failed in much the same way.  We have four of these guys in all -- one running at Molasses, one failing at Port Everglades, and the other two in my office but I haven't been able to make either of them work, I think they are both dead.

So our options are:

• reset the failsafe on the transmitter and hope it works a while longer
• allow the transmitter to fail and just get data by memory card
• replace the transmitter with a TX312
• something more exotic, maybe use the cellular modem once it's freed up

Jack visited the station on September 10th, 2010, and then reported:

I did the  transmitter reset today 9/10/10 at about 13:00 EDT.  Hope it works.

And indeed Jack's action was successful, as I confirmed by email later that same day:

It's transmitting, at least for now.  It's got two full transmissions so far.  I'll cross my fingers and hope it lasts a while!

some transmission problems

In an email written May 27th, 2010, I wrote:

I've been updating my data spreadsheets and I noticed that the Port Everglades station has been dropping more satellite transmissions lately.  The problem is intermittent and from the diagnostics it is related to GPS acquisition timeouts.  So either the GPS subsystem is failing in the old SAT-HDR-GOES (bad) or there's a problem with the GPS antenna itself (not as bad).

The problem seems to have begun at the end of April (on April 27th, more or less).  This doesn't coincide with any of Jack's visits and it doesn't follow very closely on the April 16th work, so it doesn't seem like it's a result of moving things around in the box.

We dropped 6 records in March (which is normal), 11 records in April, and 23 records so far in May.  So the problem might be getting worse but it's still not serious.  And you could argue that the near-real-time transmissions are of secondary importance for this site since Jack (I think) depends more heavily on the 6-minute datasets that we retrieve from the memory cards.

Anyhow, something to keep an eye on.  Keep in mind that we don't have any more working SAT-HDR-GOES units.  We do have some GPS antennae but I can't be sure if they're functional without a SAT-HDR-GOES to test on.

CT failure and swap, programming correction

A newly-calibrated CT was deployed on March 22, 2010, and within four days it had failed.  It was later discovered to have flooded and was discarded as irreparable.  It is not certain why it failed.

In any case, plans were quickly made to repeat the events of March 22nd, when the entire system was powered down so that the CTs could safely be swapped underwater.  This was scheduled for April 16th, 2010.  At the same time, a new program was updated to fix a configuration issue with the transmitter (it had been reporting on the wrong PID, a data-processing inconvenience) and add some more granularly reporting data tables (5-second and 30-second averages, maxima and minima).

This time the new CT worked fine, and the programming/transmission changes had their intended effect.

CT swapped, barometer added, programming updated

A view of the PVGF1 CT, sticking up at center with cable emerging from below.

Given that this station's first CT had been deployed on March 19th, 2009, plans were made to swap it out in March of 2010.  The CT's cable was arranged in a buried conduit running from the station's electronics box to the water's edge and then along the channel's bottom to the base of the channel marker it was mounted on.  The cable was arranged in such a way that there was not sufficient slack to bring it to the surface, dry it off, and switch sensors without powering the entire system down.  Therefore we coordinated schedules to make sure we had who we needed on site to safely power down the station and verify that it had started correctly afterwards.

We also took this opportunity to add an analog barometer to the system, to settle any questions about the reliability of the barometric pressure reports from the Vaisala WXT ("weather transmitter").  And the programming was enhanced to begin storing 1-minute averages, minima and maxima as well as adding more parameters to the 6-minute data table.

In an oversight, the program was deployed with a satellite transmission platform id (PID) which has been used to test the program at AOML before deployment.  This proved to be a minor inconvenience for data processing and was corrected at the next programming update a month later.