port everglades site visit

[Summary: the Port Everglades CREWS station is back online and its feeds have been reenabled.  As of this writing I expect the problem has been resolved, at the cost of no data being captured between 200 UTC April 20 2015 and 1900 UTC April 23 2015.  A more detailed narrative of the problem and fix follows.]

This morning I met Jack Stamates and Natchanon Amornthammarong (Mana) at Port Everglades for a coordinated visit to the Naval base that hosts our Port Everglades station.  They wanted to scout out possible sites for deploying Mana's ammonia sensor, and since Jack was already coordinating with the Navy for permission to access the site, I decided to tag along to work on the CREWS station.

The reason I wanted to visit was this (described here): at 200 UTC on April 20, 2015, the PVGF1 station had stopped producing data.  The station's cellular connection was working fine and it was still possible to connect to the datalogger directly.  The problem appeared to be with the datalogger's clock: over multiple connections and attempts to reset the clock, it would spontaneously revert to garbage timestamps, once in 1930 and one in 2066.  The logger's measurement operations appeared to be working normally but when it tried to write those measurements in its data tables it seems those invalid dates caused errors and no data could be saved.

On Tuesday I tried resetting the clock to the correct UTC time, but it either reverted to garbage timestamps immediately or within 24 hours.  I tried uploading variations on the logger program, theorizing that perhaps there was a problem with the memory module or memory card, but they did not help.  Rather than continue remote troubleshooting I decided to swap out the datalogger in person, figuring that this would be a brute-force solution.  I also disabled feeds of data from this station in case those 1930/2066 timestamps somehow got reported.

Since I'd be visiting the site in person, I decided to replace the GOES transmitter and WXT (the 'weather transmitter' by Vaisala) while I was there.

This site uses one of our ancient SAT-HDR-GOES transmitters, although with a direct cellular modem connection available we do not depend on GOES for data transmission.  Still, it's nice to have redundant streams of data from the same site and the GOES transmitter at PVGF1 hasn't worked properly for at least five months.  I wasn't sure if any of our remaining SAT-HDR-GOES transmitters still worked but I chose the most likely candidate for swapping, figuring at worst we'd still have only the cellular data feed, which is extremely reliable.

The WXTs at our regular ocean-based sites are swapped out annually and often they fail, at least the acoustic wind sensors do, before their year is out.  I think that's mostly because of the very large boobies that perch on our ocean-based stations, most particularly the one at La Parguera, PR.  But the last WXT at Port Everglades was deployed for almost 2.5 years with no apparent data degradation.  Its successor WXT has only been deployed for 13 months, but since we'd be on-site and it was nominally past our usual yearlong deployment time I decided to swap the WXT as well, as a low-priority task if everything else went well.

And everything did go well.  Despite a little bit of rain in the area I was able to swap the logger, transmitter and WXT.  Jack and Mana stuck around until I was done and kindly spotted me on the ladder when I swapped the WXT.  I connected to the logger by laptop before leaving to make sure all of the sensors were connected properly, and everything seemed okay, including the logger clock.

While I was connected on-site I had my first real hint of what may have caused the problem.  In looking over the variables in logger memory I was reminded that the GOES-transmitting stations actually reset their logger clocks using GPS-sourced time, once per day, as a way to avoid clock drift.  So for the first time it occurred to me that the problem might not be a logger clock failure, but rather a mangled GPS time from the transmitter.  Still, I was replacing both logger and transmitter so it didn't seem important which one had caused this problem.

I packed everything up and, after stopping briefly at home to check whether the cellular modem was still connecting (it was), returned to the lab.  Once here I connected to the station again and was surprised and a little dismayed to find that the new logger's clock was now set to a date in 1930.  So clearly this wasn't a simple case of equipment failure as I'd assumed.

What I think now is there might be some kind of date-overflow in the SAT-HDR-GOES transmitter's internal date representation, that might potentially be affected both of our transmitters (and maybe all of them).  The SAT-HDR-GOES has long since fallen out of support, and actually so has its replacement transmitter, the TX312.  We used this transmitter model at Port Everglades originally as a cost-saving measure (rather than pay for new transmitters).  It's possible that this transmitter model, which was never expected to still be operational in 2015, has lost the ability to report GPS timestamps accurately.  I theorize that resetting the logger clock with all zeroes led to the dates in 1930, and perhaps setting the clock with all-fill times could produce the dates in 2066 that I saw.

The situation now is that I've disabled the clock-reset code in the datalogger program, and instead I've manually set the time at this station based on our loggernet server time.  I have reenabled the feeds to NDBC, our CHAMP database, the CHAMP Portal, and the G2 Ecoforecasting system.  We'll be monitoring this station's performance extra-closely in the coming days but (1) the problem seems to have been fixed and (2) as a bonus we have deployed a fresh WXT that need not be swapped again for at least another year.





(signed)
Mike Jankulak

port ev logger problems

[This is the text of an email that I wrote Tuesday morning, April 21, 2015.  This blog post will be backdated to the date and time of that email.]

As it happens I received notice from NDBC this morning that they have not received any new Port Everglades data from us in over 24 hours, and there are backlogs of cronjob errors and a download alert in my mailbox today as well.

As near as I can tell, it looks like a problem with the datalogger's internal clock.  I am seeing timestamps in the year 2066 and a few times when I've connected the internal clock has been reset to 1930.  I have reset the logger clock a few times to the correct time (UTC).  The first few times it got corrupted again almost right away.  At present it has held on to the correct time for about ten minutes but I don't trust it.

[These bad timestamps are causing the logger to skip its datatables.  Essentially I think its measurement actions are working just fine but it balks when asked to record those measurements with out-of-bounds timestamps.]

Anyhow I have disabled all of our PVGF1 feeds for the present.  In the short term it may be possible to reenable them by day's end if the clock seems to be holding steady.  In the long term we will probably want to replace that logger.

I know Jack and Mana might be visiting that site soon so depending on their timing I might suggest that I tag along, if that's alright with everyone.

(signed)
Mike Jankulak

station online and fully operational

The PVGF1 station as seen in 2009.

This is meant to serve as a status report for the Port Everglades CREWS station.  It was installed on March 19th, 2009 as a collaborative effort between the Florida Area Coastal Environment (FACE) program and the Coral Reef Early Warning System (CREWS) technology developed by the Coral Health and Monitoring Program (CHAMP).  Both programs operate out of NOAA's Atlantic Oceanographic and Meteorological Laboratory (AOML).

The station ran for two and half years before its ADCP (Acoustic Doppler Current Profiler) stopped communicating.  This led to the removal of the station's underwater instruments (the ADCP, and a CT) on October 20th, 2011 and to the effective end of FACE's role in the project.  After this, the station continued to lead a second life as a meteorological-only, CREWS-only station, now connected by a cellular modem which gives us on-demand access to its detailed data reserves.  For example, the analog air temperature and pressure sensors are sampled every five seconds and each of those 5-second readings are stored separately in memory and downloaded to CHAMP servers every five minutes.  It is also possible to access and update the station's programming at any time via the cellular modem.

Meanwhile the station's SAT-HDR-GOES transmitter appears to be failing.  This satellite transmitter was the station's original means of communications when first deployed in 2009.  Its reliability was decidedly uneven until October of 2011, when two things happened:  one, we swapped one transmitter for another of the same model, and two, we deployed the cellular modem.  These two means of communications continued to operate fully redundantly until August of 2014, when there started to be a large number of transmission problems on the satellite side.  However, these problems amounted to little more than an intellectual curiosity since the cellular modem continues to supply all of our connectivity needs and then some.  The satellite transmitter, which is essentially obsolete in this configuration, continues to experience problems as of this writing and at some point our processing will be updated to simply ignore it.

The station is now five and a half years old and continues to provide significant meteorological data as well as act as an important test site for CHAMP cellular communications.  Its underwater experiments could be easily resumed at any time if the need/interest arises.

WXT swapout, recovery of batteries

On March 10th, 2014, Mikes Shoemaker and Jankulak met up at the Port Everglades CREWS station.  Our goal was to swap out the Vaisala "weather transmitter," or WXT, which is an instrument that reports air temperatures, pressures, winds, gusts, relative humidity and precipitation.  This sensor was deployed on October 20th, 2011, about two and a half years previously.  Although it showed no signs of data degradation, we still thought it should be replaced since our practice in the considerably harsher marine environments of other CREWS stations is to swap out the WXTs after only one year of deployment.

Mike Shoemaker also wanted to take the opportunity to recover some of the batteries and chargers that had been installed in 2009 during this stations initial deployment.  There were originally four batteries and four chargers installed; one battery/charger provided a reliable power feed to the datalogger and met package and continuity of power even on those occasions when the external FL&L power supply went out.  The other three batteries were connected serially to deliver the higher voltages required by the ADCP.  With the removal of the ADCP in October of 2011 and no plans at present to redeploy it, Mike S. judged that three of the station's four batteries/chargers could potentially be reused elsewhere.

We spend the morning at the station, removed three batteries and chargers, replaced the WXT, and rearranged the equipment that remained.  All systems were confirmed to be operational before leaving.

FACE portion of project concludes, station goes live with new met package and cellular modem

Things got rolling on the plans to remove this station's ADCP and CT with a meeting on the morning of September 21st, 2011.  Jack wrote this summary of what was decided:

As the ADCP cable and the CT cable are married together with tape and bio-fouled together underwater it makes it unlikely that they can be separated underwater.

What we plan to do is remove these cables but leave in place a "traveler" (a pulling line) so that, at a later date, we can install a new cable through the existing cable path.  Then, a CT or any other instruments can be deployed in the inlet.

The meteorological senors will remain in place ( perhaps be swapped out with  fresh ones) and continue to transmit data.

We hope to do these operations on Oct 19th.

As the operations date approached, however, the weather turned bad and the boat operations could not safely take place on the appointed date.  The operation was therefore divided into two parts, a land-based component (which was moved from the 19th to the 20th) and a sea-based component that could be delayed until the following week.  Jack summarized the change in plans on October 17th, 2011:

The weather precludes boat ops this week.

The shore team will try and go out on Thursday if the weather is better and cut the cable in two parts, the land side and the wet side.  We will recover the land side and sink the end of the wet side with a weight.

Shore-based operations did indeed go as planned on October 20th, 2011.  Later that afternoon I sent out the following report by email:

Just back at my desk now, and the quick report is that everything went well this morning at the port.

The GOES transmissions have resumed, this time on a 300-baud channel since the 100-baud channels are going away next spring.

We have also deployed the Raven AT&T cellular modem.  So now we have full access to that datalogger, we can download data (not just the hourly stuff that comes by GOES, but 6-minute, 1-minute and 30-second averages too), we can upload/download new logger programming, and we can connect to the station from the lab and watch the 5-second data updates flash past in real time.

We swapped out the "weather transmitter" or WXT, the Vaisala device that measures winds, temperatures, pressures and precipitation.  The old one was still producing (what appeared to be) reliable data but we normally deploy those puppies for 1 year and that one has lasted 2.5 years.

I also added a standalone air temperature sensor so (along with the barometric pressure sensor) we now have redundant measurements of air temperatures and pressures from multiple instruments.

The underwater instruments are now disconnected from both power and communications, so there will no longer be any salinities or sea temperatures reported from this station (at least not until we decide to deploy something new here!).

I will be patching up our feeds to NDBC and G2, and parsing out the data from the card that I collected this morning.  That will hopefully be online for everyone by tomorrow afternoon.

 With this operation, the Port Everglades station effectively ceases to be a FACE/CREWS hybrid since all of the FACE elements have now been removed.  Jack has handed over to me the "key" that is used to open the datalogger box on base.  All underwater instruments are disconnected, although as noted we have taken steps to make it easy to redeploy them, should there ever be interest in doing so.

Also worthy of note is that this station now becomes only the second CREWS station (after Saipan) to make use of a cellular modem.  The station continues to transmit by satellite on its new 300-baud platform, via a swapped-out transmitter that was recently recovered from another "hybrid" station at Molasses Reef.  The change in transmitters appears to have fixed the frequent problems that have required Jack to push the transmitter's failsafe-reset button during many of his data-download visits.  However, the presence/lack of transmitter problems is rendered effectively moot by our constant and super-reliable cellular connection to the station, and there is no longer any reason to manually recover the datalogger memory cards on a regular basis.

In-person visits to this station are expected to take place far less frequently from now on, and without any FACE involvement.

station goes offline, and is revived (redux)

On July 21st, 2011, we stopped by the station by boat on our way to visiting some other FACE ADCP sites, and Jack jumped in to have a closer look at the ADCP and CT.  Partly he wanted to see if there were any obvious explanations for the ADCP's loss of communications, and partly he wanted some context for drawing up a plan to remove both sensors later on.  Once we returned to the lab, I noticed that the station's transmitter had failed again only the afternoon before.  I sent out this update:

Weird timing on this one -- I didn't realize while you were examining the Port Everglades CT and ADCP this morning that the station had stopped transmitting yesterday afternoon.  I guess there wasn't anything we could have done about it today anyhow even if we'd known, not without clearance from the Navy to approach the box on land.

The station's last transmission was at 1800 hours UTC yesterday, or 2pm local time.

From the diagnostics it looks to me like another transmitter failure.  Doing the failsafe reset might bring it back online.  Another option is to try swapping it with the SAT-HDR-GOES transmitter I recently recovered from Molasses Reef, which was still functional at the time of its recovery.

Jack returned (by land) to Port Everglades on August 8th, 2011 and his transmitter reset brought the station back online as I reported by email:

Jack made it out to Port Everglades this morning and punched the reset button on the transmitter.  In response the station has made one transmission so far (that's all it's had time for) so I am cautiously optimistic that we are live again, at least for the time being.

I later followed up with on August 10th, 2011, with more details about the diagnostics reported by the transmitter while offline:

I've already mentioned this to Jack, but in case anyone else is interested in the Port Everglades data, it's all been extracted and that missing month of data has been patched.  Files and spreadsheets are available in the usual places, feel free to ask if you want a pointer to anything specific.

And yes, the data show that the transmitter really was in failsafe mode.  This was possibly caused by an excess of GPS-acquisition-related errors in the time leading up to the failure.  As of this writing the station has been transmitting okay for just over two days in the time since Jack's visit.

data downloads, 2011

Between an on-site intervention to bring the satellite transmitter back online on January 13th, 2011 and this station's significant swapout/reinvention on October 20, 2011, FACE continued their routine of (roughly) monthly visits for downloading the ADCP's data reserves and swapping datalogger memory cards.  These visits gave us semi-regular access to ADCP data, which weren't included in the station's near-real time transmissions, allowed us to "patch" holes in our archives where occasional transmissions had been dropped, and gave us access to the more time-granular (5-second, 30-second, 1-minute, 6-minute) stored data that weren't included in the hourly transmission.

Between January 13th and October 20th, 2011, these data-download visits took place on:

• February 2nd, 2011
• March 4th, 2011
• May 17th, 2011
• June 30th, 2011
• August 8th, 2011