Thursday, October 20, 2011

FACE portion of project concludes, station goes live with new met package and cellular modem

Things got rolling on the plans to remove this station's ADCP and CT with a meeting on the morning of September 21st, 2011.  Jack wrote this summary of what was decided:
As the ADCP cable and the CT cable are married together with tape and bio-fouled together underwater it makes it unlikely that they can be separated underwater.

What we plan to do is remove these cables but leave in place a "traveler" (a pulling line) so that, at a later date, we can install a new cable through the existing cable path.  Then, a CT or any other instruments can be deployed in the inlet.

The meteorological senors will remain in place ( perhaps be swapped out with  fresh ones) and continue to transmit data.

We hope to do these operations on Oct 19th.
As the operations date approached, however, the weather turned bad and the boat operations could not safely take place on the appointed date.  The operation was therefore divided into two parts, a land-based component (which was moved from the 19th to the 20th) and a sea-based component that could be delayed until the following week.  Jack summarized the change in plans on October 17th, 2011:
The weather precludes boat ops this week.

The shore team will try and go out on Thursday if the weather is better and cut the cable in two parts, the land side and the wet side.  We will recover the land side and sink the end of the wet side with a weight.
Shore-based operations did indeed go as planned on October 20th, 2011.  Later that afternoon I sent out the following report by email:
Just back at my desk now, and the quick report is that everything went well this morning at the port.

The GOES transmissions have resumed, this time on a 300-baud channel since the 100-baud channels are going away next spring.

We have also deployed the Raven AT&T cellular modem.  So now we have full access to that datalogger, we can download data (not just the hourly stuff that comes by GOES, but 6-minute, 1-minute and 30-second averages too), we can upload/download new logger programming, and we can connect to the station from the lab and watch the 5-second data updates flash past in real time.

We swapped out the "weather transmitter" or WXT, the Vaisala device that measures winds, temperatures, pressures and precipitation.  The old one was still producing (what appeared to be) reliable data but we normally deploy those puppies for 1 year and that one has lasted 2.5 years.

I also added a standalone air temperature sensor so (along with the barometric pressure sensor) we now have redundant measurements of air temperatures and pressures from multiple instruments.

The underwater instruments are now disconnected from both power and communications, so there will no longer be any salinities or sea temperatures reported from this station (at least not until we decide to deploy something new here!).

I will be patching up our feeds to NDBC and G2, and parsing out the data from the card that I collected this morning.  That will hopefully be online for everyone by tomorrow afternoon.
 With this operation, the Port Everglades station effectively ceases to be a FACE/CREWS hybrid since all of the FACE elements have now been removed.  Jack has handed over to me the "key" that is used to open the datalogger box on base.  All underwater instruments are disconnected, although as noted we have taken steps to make it easy to redeploy them, should there ever be interest in doing so.

Also worthy of note is that this station now becomes only the second CREWS station (after Saipan) to make use of a cellular modem.  The station continues to transmit by satellite on its new 300-baud platform, via a swapped-out transmitter that was recently recovered from another "hybrid" station at Molasses Reef.  The change in transmitters appears to have fixed the frequent problems that have required Jack to push the transmitter's failsafe-reset button during many of his data-download visits.  However, the presence/lack of transmitter problems is rendered effectively moot by our constant and super-reliable cellular connection to the station, and there is no longer any reason to manually recover the datalogger memory cards on a regular basis.

In-person visits to this station are expected to take place far less frequently from now on, and without any FACE involvement.

Wednesday, August 10, 2011

station goes offline, and is revived (redux)

On July 21st, 2011, we stopped by the station by boat on our way to visiting some other FACE ADCP sites, and Jack jumped in to have a closer look at the ADCP and CT.  Partly he wanted to see if there were any obvious explanations for the ADCP's loss of communications, and partly he wanted some context for drawing up a plan to remove both sensors later on.  Once we returned to the lab, I noticed that the station's transmitter had failed again only the afternoon before.  I sent out this update:
Weird timing on this one -- I didn't realize while you were examining the Port Everglades CT and ADCP this morning that the station had stopped transmitting yesterday afternoon.  I guess there wasn't anything we could have done about it today anyhow even if we'd known, not without clearance from the Navy to approach the box on land.

The station's last transmission was at 1800 hours UTC yesterday, or 2pm local time.

From the diagnostics it looks to me like another transmitter failure.  Doing the failsafe reset might bring it back online.  Another option is to try swapping it with the SAT-HDR-GOES transmitter I recently recovered from Molasses Reef, which was still functional at the time of its recovery.
Jack returned (by land) to Port Everglades on August 8th, 2011 and his transmitter reset brought the station back online as I reported by email:
Jack made it out to Port Everglades this morning and punched the reset button on the transmitter.  In response the station has made one transmission so far (that's all it's had time for) so I am cautiously optimistic that we are live again, at least for the time being.
I later followed up with on August 10th, 2011, with more details about the diagnostics reported by the transmitter while offline:
I've already mentioned this to Jack, but in case anyone else is interested in the Port Everglades data, it's all been extracted and that missing month of data has been patched.  Files and spreadsheets are available in the usual places, feel free to ask if you want a pointer to anything specific.

And yes, the data show that the transmitter really was in failsafe mode.  This was possibly caused by an excess of GPS-acquisition-related errors in the time leading up to the failure.  As of this writing the station has been transmitting okay for just over two days in the time since Jack's visit.

Monday, August 8, 2011

data downloads, 2011

Between an on-site intervention to bring the satellite transmitter back online on January 13th, 2011 and this station's significant swapout/reinvention on October 20, 2011, FACE continued their routine of (roughly) monthly visits for downloading the ADCP's data reserves and swapping datalogger memory cards.  These visits gave us semi-regular access to ADCP data, which weren't included in the station's near-real time transmissions, allowed us to "patch" holes in our archives where occasional transmissions had been dropped, and gave us access to the more time-granular (5-second, 30-second, 1-minute, 6-minute) stored data that weren't included in the hourly transmission.

Between January 13th and October 20th, 2011, these data-download visits took place on:
  • February 2nd, 2011
  • March 4th, 2011
  • May 17th, 2011
  • June 30th, 2011
  • August 8th, 2011

Monday, July 18, 2011

CT reliability questioned, but a swapout is canceled when the ADCP fails

On May 18th, 2011, Jack Stamates (FACE project) circulated the following update, including some early concerns about a possible failure of the station's CT:
I went to the Port yesterday.  I did the transmitter reset about 17:30 UT.  I swapped out the card and it is pinned to your door.

Tomorrow (5/19) the Hildebrand will try and make a salinity measurement in the vicinity of the instrument to see if the sal sensor is going bad.

I hate to say it but my gut feeling is that the sensor is failing.  It has been too elevated for too long now.  But I will remain hopeful that I am wrong.
I replied on May 20th, 2011 with some context about the CT's deployment duration:
I hope the CT is still okay but it's not too surprising if it's fallen out of calibration, normally we aim to swap out CTs and CTDs once a year and this one is overdue by that standard (deployed April 15th, 2010).
In part because of my input about a CT's usual one-year deployment lifetime, and in part because of Jack's lingering concerns about the reliability of the CT, plans were made to do another CT swap at the station, targeted for July.  This would again require powering down the station completely so that the CT could safely be unplugged and swapped underwater.  I meant to take this opportunity to update the station programming with some code developed for the other CREWS stations.  As I explained in email on June 23rd, 2011:
I've volunteered to be onsite at Port Everglades for this CT swap.  The reason is that I'd very much like to update the PVGF1 logger program to add in the CF Memory diagnostics that I recently released to the CREWS station in St. Croix (and now also at Molasses Reef).  Basically this would give me peace of mind regarding the station's local memory logging, instead of the minor anxiety I get whenever Jack brings me one of the memory cards to download (and I'm unsure until I plug it in whether I'll find that it's logged any data correctly).  We've never yet had a problem at Port Everglades but we have twice had failures at other CREWS stations and I'd prefer to be able to monitor the local memory collection remotely (which is what this programming update will allow me to do).
These plans were shelved, however, when we lost contact with the ADCP at Port Everglades.  Since the CT data were collected specifically to support the analysis of the ADCP data, our focus now changed from a CT swap to a complete removal of the underwater instruments at this site.  As Jack mentioned in email on July 5th, 2011:
The ADCP in the Port Everglades channel is, as far as I know, not going to be serviceable.  This means we will NOT be swapping the CT sensor out.

On 7/21 I hope to change out the Broward FACE ADCP.  (The ADCP that is off shore near Hillsborough.)

The only thing I am thinking about doing in the Port Everglades channel is jumping in to have a quick look to see if there is anything obvious and to get an idea about what to expect when we do do the system removal.
On July 18th, 2011 it occurred to me that we were still feeding salinity data to NDBC, and I asked Jack by email whether the data were unreliable enough that we should stop doing this:
I checked our NDBC feed, and we are sending them Port Everglades data from the CT.  We send water temperature, salinity and depth.  The depth is hardcoded to 1.65m, my code comments say this figure was one that you provided.

Anyhow, do you think we should stop sending them any data from the CT? Or should we hold back the Salinity data but continue to send them water temperature?
Jack replied that same day that the CT salinity data were indeed unreliable and should no longer be trusted:
I just plotted the last data you sent me.

During the times when the CT Sal  has discontinuities,  the CT temperature tracks the ADCP temp closely so it think temperature  it is OK but the Sal is definitely "out to lunch."

I think we should  stop sending the salinity.  At some point in the near future I will need to remove the ADCP but not this month.

Wednesday, April 27, 2011

salinity blips

On March 24th, 2011, while updating my data spreadsheets, I noticed that there had been a large jump in salinity on March 15th, which I called to Jack's attention in an email as follows:
I was updating my spreadsheets again today and I noticed that the Port Everglades station's salinity numbers had a bounce on March 15th (see attached graph showing data from January 1st through today), and I thought you might be interested.

There doesn't seem to be any associated rain event (and anyway I imagine that would cause a drop, not a rise) so my guess is someone was cleaning the CT?  Or maybe something else was going on.
The x-axis is the day of the year, showing data from the beginning of 2011.  The y-axis is salinity measured in PSU.
Then on April 11th, 2011, I was again reviewing the station's recent data and commented on both the transmitter diagnostics and a second salinity "blip":
I'm updating my data spreadsheets this morning and I happened to notice that the transmitter diagnostics from Port Everglades are showing signs of trouble with GPS acquisitions again.  I'd like to suggest that you perform the transmitter's "failsafe reset" procedure again the next time you visit to see if that helps.  There does not appear to be any impact on transmitter performance at this point but I'd like to see if the reset will clear up the GPS problems before they start affecting transmissions.

Also, in addition to the salinity "blip" that I told you about from March 15th, there seems to have been another blip yesterday, April 10th.  Just fyi.
Jack replied on April 27th, 2011, with an information that about a probably connection between these "blips" and cleaning visits from the divers:
I got info back from the divers.  They cleaned the instrument on 3/16 (possibly 3/18,  they were not positive) and on 4/21.

Thursday, January 13, 2011

station back online

[The following is a slightly edited version of an email I sent to Jim Hendee (CHAMP principal investigator) on January 13th, 2011.  It summarizes the results of a visit by Jack Stamates (FACE project) and myself to the station that morning, where our intention was to investigate what might have caused the station to stop transmitting on December 29th, 2010.]

On January 13th, 2011, I wrote:
Jack and I visited the station this morning.  The entire base (?) lost power for a week over new year's, Dec 27th to Jan 3rd.  [This was apparently due to a disastrous FP&L screwup.  Jack knows/understands more of the details of this.]

The station batteries performed beautifully and everything continued to operate normally (ADCP, CT, logger, WXT) except the sat transmitter.  So we lost no data whatsoever.  Jack was thrilled with this test of the station's operation during an extended power loss.  Great design!

The one weak point continues to be the ancient satellite transmitter.  At the battery levels recorded by the logger, even the transmitter should have continued to operate normally.  But something about those slightly-lower voltage levels, or possibly there were repeated voltage drops or spikes, something spooked the transmitter.  It was NOT in failsafe mode (I could see that when I connected directly to the transmitter) but it had stopped communicating with the logger, so it had no data to transmit.  Pushing the failsafe-reset button seems to have brought it back to life.  Remember, this is one of those old SAT-HDR-GOES transmitters, the last one (of five original) that still works at all.

Mike J+
Note that while Jack and I were visiting, we again downloaded all available ADCP data and swapped datalogger memory cards, an operation that normally occurs about once a month.

Wednesday, January 5, 2011

transmitter failure

[The following post is a copy of an email message I wrote to Jack Stamates on January 5th, 2011.  Transmissions from the Port Everglades station had stopped on December 29th, 2010, and we were planning to visit the site to see what may have gone wrong.]

On January 5th, 2011, I wrote:
I've updated all of my spreadsheets and data files with the most recently retrieved data card, from November 9th.  I've also had a closer look at the data from immediately before the station stopped transmitting, and there's a suggestion that something weird started happening a few days before.

The first hint of trouble was a skipped transmission on Monday, December 27th at 1pm local time.  When transmissions resumed in the next hour, the voltage diagnostics looked different than before.  The logger reports min, max and average voltage numbers -- normally the min numbers hang around 12.8V, the average at 13V, and the max around 13.8V.

Beginning Monday afternoon, the max and average value both dropped to about 12.8V and the min fell below 12.5V.  Things went on like that for about two days until transmissions ceased on Wednesday night (Dec 29th) at about 8pm local.

So this probably isn't the same transmitter-failure problem we've seen in the past.  The sure sign of that problem is that the GPS acquisition times go through the roof in the times leading up to the failure, and there was no sign of that in this case.

It would seem the station has a power issue of some kind.  This could mean that something weird is going on with the external power source; or the datalogger might be suffering some kind of power problem internally; or something connected to the logger might have short-circuited.  [In the past, we've seen something similar when an underwater instrument has a bulkhead failure and floods, shorting power to ground and draining the power supply of all the other instruments.]  There is no sign of trouble with the CT numbers that I can see in those past few days, or the meteorological instruments.

So anyhow, I'd suggest you prepare yourself for a wider range of problems when you go.  If you want me to meet you there, let me know, it's usually no trouble since I live so close.  Also, there's no guarantee that this is a transmitter-only problem, so you may find that the logger hasn't been storing the CT data to its memory card in the last week.

I've formatted your spare memory card so you can swap them out during your next visit.  I'll bring in by your office today.

Mike J+