In the previous 48 hours, I was able to code the scraping routines for CWAADS (Coastal Weather Anomaly Advance Detection System). I had a few tweaks to perform in transitioning it from manual triggering to a CRON trigger. Mainly, I incorrectly assumed that CRON would recognize the path to PHP and it didn’t. Once I hard-coded the path to the called PHP routines, the data started pouring in.

Currently, at 20 and 50 past the hour, CWAADS is pulling in 1066 records from data buoys from around the globe.

The data is pulled from RSS feeds provided by the National Data-Buoy Center at ndbc.noaa.gov. The data generators maintain any and all copyrights pertinent to their said data. JayCTheriot.Com Enterprises maintain copyrights on the CWAADS process and data visualization products. The unexpurgated data is then stockpiled for later analysis and data visualization.

A MySQL database, running on a Ubuntu 20.04 headless container instance running on a Dell R420 1U 24-core server using Ubuntu 20.04 as the OS, is the data’s indefinite home.

Currently the data structure is simple as the desire was strong to begin collecting data. There is an autoincrement, ID and longtext field storing the data. In the close future, the data will be mapped and stored in a multi-field base attributing each field to a data value from the respective buoys. As no buoy reports the same datafields, and my body was failing me, I sought to simplify the collection process.

The collected data can then be searched using the “LIKE” clause and wild cards. For example:

SELECT * from bd_data like "%Wind%";

will select all the records that contain Wind Speed, Wind Gusts and other data points relating to wind. The method is horribly crude. My next step is to map the incoming data to appropriate field titles. Thus, the current roadmap resembles:

1. Map incoming data to conveniently named fields.
2. Create GMT (Generic Mapping Tools) container instance to handle the data visualization (DV)
3. Create public-looking web-fronts to display the DV products for the public to access them.
4. Create a method for the public to comment and/or suggest improvements
5. Maintain and improve the analysis, appearance and availability of data and datasources.

Thanks for reading…more to come,

Jay C. “Jazzy J” Theriot