Data Flow
From Seamonster
- Data main page
- Database description
- Data Flow
- Gnomes
- SQL Server
- Visualization, Photosynth, Time Lapse Videos, Chart Director, Virtual Globes, Virtual Earth, Google Earth
- XML, SensorML, GML, KML
- Web Services
- Geoserver, PostGIS
Contents |
Introduction
This page acts as a Tutorial so that a New Monster can learn how the data flows through the system, bit by bit.
Campbell to VuS to Database
This is the data flow that we would like to eventually establish on all our met stations. Right now this is up and running for the Upper Lemon Glacier.
Campbell Data Logger (CR1000)
We are using the CR1000 for our met station. The Upper_Lemon_Met.cr1 program running on it was created in shortcut and then adjusted to stream the last four table values every minute. So as long as the VuS reads the data once every hour, we should be getting uninterrupted data flow.
Microserver 54
This is the VuS that is attached to the CR1000 through a USB cable. These are the crontab entries:
0 * * * * /root/brickmove/bin/housekeeping.sh >/dev/null 2>&1 1 * * * * /root/brickmove/bin/uS_data_router.py >/dev/null 2>&1
This just means that at the top of every hour (0 * * * *) the housekeeping.sh script is called and on the first minute of every hour the uS_data_routing.py script is called.
housekeeping.sh
This script goes through the following steps:
- Checks if there is a process id in the 'brickmove.pid' file, if so it kills that process.
- Checks if there are any '.log' files in the 'logs' directory, if so it moves them to the 'inbound' directory.
- Then it starts to listen (cat) to the USB port (/dev/ttyUSB0) and appends everything to a '.log' file.
- The process id of this process is written to the 'brickmove.pid' file.
The killing of the process in the beginning ensures that the data is recorded to a new file every time the script is run. So in this set up we get a new data file every hour.
uS_data_routing.py
This program is designed to route data from the VuS back to the primary SEAMONSTER server. The program works in the following manner: A function has been defined for each uS and when the program is placed on a machine it will determine which function to call based on parsing its IP address.
- The program first looks to see if there are '.log' files in the 'inbound' directory and, if so, selects the file from the earliest date and appends the other files in the directory to that file.
- All files except for the appended file are then moved to the 'expired' directory.
- After clearing the 'inbound' directory of all but one file the program attempts to ping the SEAMONSTER server. If the ping is successful the file in the 'inbound' directory is sent via rsync and then moved to the 'expired' bin. If unsuccessful, the VuS attempts to ping the VuS closest to the server and if successful will rsync the data there instead.
