CTD Database Development Notes
Please note as of mid-2015, a revision of the CTD database is currently underway. With additional data quality code columns added to the CTD+bottle csvs offered as final processed data, new files are being created by re-processing the CTD archives. Measurement values should remain unchanged but the format is being updated.
Older information about database development is described below:
Starting with CalCOFI 0907M2, the BTLvsCTD program (developed by SIO-CalCOFI) generates several database-friendly csvs. With Seasave Data Processing software v7.19, sensor calibration coefficients are embedded as xml data in the .hdr file. BTLvsCTD.exe parses the .hdr for all stations into three csvs: hdr.csv, xmlcoeff.csv, & span.csv.
- The .hdr csv includes the cast information such as date, time, line, sta, lat & lon, sensor configuration, and all Seasave data processing modules and coefficients applied.
- The span.csv is a tabulation of the min and max values for each sensor, extracted from the .hdr file. This file/table will be useful for automating basic data-quality controls and sensor behavior assessments, allowing data processors to query data ranges for validity.
- The last file is xmlcoeff.csv, a tabulation of the coefficients for sensor.
Starting with 0907, the first cruise to use dual SBE43 O2 sensors so the data tables will have these columns by default. Earlier cruises, using single O2 sensors will auto-fill the tables, leaving the second O2 columns blank.
Since sensor installation may vary from cruise to cruise and sensors may occupy different voltage channels, the voltage references v1 v2 v3...
are replaced by TransV FluorV AltV... whenever possible.This requirement required a revision of BTLvsCTD.exe dated 9 Nov 2009 since I had originally maintained the voltage # in respect to sensor. But this creates a database import problem; v0_Trans, v1_Fluor will not integrate with v0_Fluor, v1_Trans so both are renamed TransV FluorV, removing the specific voltage # reference.
Currently, relabeling of the voltage channels is offline since it creates data reprocessing issues. You will have to refer to the hdr file to determine which sensor occupies the 8 voltage channels (V0-V7). We try to keep the channels consistent unless deploying test sensors for performance assessment. Like the RINKO Oxygen Optode on 1210NH.
- V0 = Transmissometer
- V1 = Fluorometer
- V2 = PAR
- V3 = Altimeter
- V4 = Primary Oxygen
- V5 = Secondary Oxygen
- V6 = ISUS Nitrate
- V7 = pH
- V15 = Surface PAR, when made available by the vessel; SIO's was retired
BTLvsCTD.exe's main module also generates bottle-corrected upcast and downcast csvs, merging all the individual station up and down csvs into two files. A dbcoeff.csv file is generated tabulating the cruise-ave bottle vs CTD coefficients as well as the station regression coefficients (each station's CTD data are regressed against bottle samples from that station, reducing the station-to-station variability witnessed using the cruise-ave coefficients). If bottle data are not available then only cruise-ave coefficients are used to correct the CTD data.
BTLvsCTD.exe has a module to automatically relabel the v0 v1 v2... to the sensor voltage in all .asc files. (I plan on using the hdr to automate this but currently the table in BTLvsCTD tabulating voltages vs sensor is used.)
BTLvsCTD.exe has a file renaming module that renames all dYYMM###.asc & .hdr, uYYMM###.asc & .hdr files to web-format YYMM_LLLLSSSS_###d.asc & .hdr, YYMM_LLLLSSSS_###u.asc & u.hdr.