Developers
Alert & Push Streaming
  • Reference
  • Examples
  • Demos
  • Introduction to Alert & Push

    Synoptic Alert and Push Data Services (Beta v0.5)

    The data alert and push services allow users to “plug” into the real-time data streams after data checks have been applied. Roughly 25 million observations flow through this system every day, and these data services allow for the filtering of various domains and observed surface conditions from 100’s of mesonet networks and over 30,000 active stations.

    See this service in action on the demos page

    Connection and Data Format Details

    Establishing Connections

    The connection protocol uses secure websockets (wss://). Establishing a connection to the data push services requires a wss connection to our server enpoint. The connection arguments can take one of two forms: initial connections and reestablishing connections.

    Initial Connections

    wss://push.synopticdata.com/feed/{valid_api_token}/?{optional_arguments}
    

    This form expects a valid API token and any optional arguments to limit the selection domain. For example:

    wss://push.synopticdata.com/feed/{valid_api_token}/?state=ca&vars=air_temp,wind_speed

    In this case, all observations originating from stations within the state of California that report air temperature or wind speed will send to the established connection within seconds of the observation entering our data ingest system.

    The initial response will take this form

    {
        "code": "success",
        "messages": ["Starting new session."],
        "session": "993624a1-2563-412e-9cfb-647535a364ac",
        "type": "auth"
    }
    

    or

    {
        "code": "failed",
        "messages": ["invalid token"],
        "session": "",
        "type": "auth"
    }
    

    Re-establishing Connections

    wss://push.synopticdata.com/feed/demotoken/993624a1-2563-412e-9cfb-647535a364ac

    Upon successful initial connections, a session ID is provided that can be used to reestablish a severed or stopped connection. This is useful to ensure all data since the last push is resumed at the exact location without missing a single observation. The use of this form will ignore all optional arguments and use the arguments supplied with the initial connection if any were given.

    The response for reestablishing connections will take this form

    {
        "code": "success",
        "messages": ["Restarting from provided session id."],
        "session": "33258f22-77b7-4514-8a30-a792e97e170a",
        "type": "auth"
    }
    

    Response Types

    All response objects returned from the data push service are valid JSON. Every JSON object will contain the key of “type” which indicates exactly what the object represents. There are 3 types.

    JSON Reponse Type Description
    auth Used to indicate the status of attempted authentication. Successful authentications will provide a session ID.
    metadata Used to indicate all information in the JSON object is metadata relating to UNITS or STATIONS.
    data Used to indicate all information in the JSON object is observational data matching the domain criteria provided for the active session.

    Stopping Connections

    Connections can be explicitly stopped at any time. Sending the command to terminate the existing websocket connection takes this form

    {
        "feed_action": "stop"
    }
    

    Stopping a connection or simply severing the network connection is effectively the same thing. If the connection is reestablished, all data not pushed since the last successful push will begin to flow.

    Data Object Response Formats

    By default, basic metadata of all data elements are also sent. Metadata will always flow prior to the first occurrence of data, which guarantees metadata is available for any given observational data object. The metadata format takes this form

    {
        "units": [
            {
                "sensor": "air_temp",
                "unit": "Celsius"
            },
            {
                "sensor": "wind_speed",
                "unit": "m/s"
            }
        ],
        "type": "metadata",
        "stations": [
            {
                "latitude": "40.76623",
                "stid": "WBB",
                "elevation": 4806.0,
                "network": 153,
                "longitude": "-111.84755"
            },
            {
                "latitude": "38.89724",
                "stid": "MOA18",
                "elevation": 829.0,
                "network": 156,
                "longitude": "-92.21807"
            },
            ...
        ]
    }
    

    Objects containing observational data will have this form

    {       
        "type": "data",
        "data": [
            {
                "set": 1,
                "stid": "KXBP",
                "value": -6.69,
                "qc": [],
                "date": 201801132215,
                "sensor": "wind_speed"
            },
            {
                "set": 1,
                "stid": "C4824",
                "value": 6.56,
                "qc": [],
                "date": 201801132220,
                "sensor": "air_temp"
            },
            {
                "set": 1,
                "stid": "C4824",
                "value": 0.0,
                "qc": [],
                "date": 201801132220,
                "sensor": "wind_speed"
            },
            {
                "set": 1,
                "stid": "D7524",
                "value": -8.33,
                "qc": [],
                "date": 201801132221,
                "sensor": "air_temp"
            }
        ]
    }
    

    The “data” entry is a list of observations broken into individual variable values. This pattern follows other data services such as the Mesonet API. Each nested object of data contain the following keys:

    Key Description
    stid The station identifier known within all Synoptic data services.
    date The date and time of the observation. This is in the form of yyyymmddhhmm (year, month, day, hour, minute) in UTC.
    sensor The sensor (variable) name the data represents. The names used are the same as all Synoptic data services (see API).
    set The instance of the sensor for the given station. Some stations may report more than one instance of a sensor, such as air temperature at 2 meters and 10 meters, or soil temperature at 10 cm below the surface and 30 cm below the surface.
    value The value of the sensed observation for that moment in time. The units will be in what was supplied on the initial connection. Default is metric. The metadata JSON object will contain all unit information for clarification.
    qc A list of all data checks and quality control flags that indicate a failed test. For example, sl_rate_change reflects a rate change test failed for the given observation.

    Web Socket Connection Arguments

    Domain Selection

    Argument Description Example
    state Provide a list of state codes to limit the observation domain to consider.
    Example state=ut,ca,nv
    network Provide a list of network IDs to consider. All other observations are ignored.
    Example network=1,65,2,153
    radius Provide a latitude and longitude with a radius in miles. Only observations originating from the radius will be considered. Alternatively, provide a station ID and a radius.
    Example radius=41.5,-111.2,50 or radius=KLAX,100
    nwszone Provide a list of NWS county warning areas as the domain to consider.
    Example nwszone=VA525,UT003
    nwsfirezone Provide a list of NWS fire zones as the domain to consider.
    Example nwsfirezone=SLC478
    bbox Provide a bounding box as the domain to listen to. The rectangle must be in the form of (min_lon, min_lat, max_lon, max_lat).
    Example bbox=-100,30,-90,40
    stid Provide a distinct set of stations to monitor. Only these stations will have data returned.
    Example stid=KSLC,WBB,KSFO

    Sensor and Data Checks

    Argument Description Example
    vars Provide a list of sensor names to limit returning only those data elements. Sensor names are in the form found on the Mesonet data API. All names are identical. The response returned will contain these names as the sensor as well. Omitting this argument will return all available sesnsors for the matching stations within the selection domain.
    Example vars=air_temp,wind_speed,wind_gust,pressure,relative_humidity
    qc A list of data checks that remove observations matching the given list. By default, no observations will be returned that have been flagged with a failed range test (sl_range_check). It is up to the user to decide what flags are important to apply to the data based on the use case.
    Example qc=sl_rate_change,sl_persistence

    Filter Conditions

    Argument Description Example
    filter_on Set of conditional expressions defined similar to a SQL where statement. Only data matching the criteria set will be returned in the feed. Each condition must be enclosed within parenthesis and have a sensor name, evaluator, and value. Conditional phrases may be chained together with “and” or “or” operators. Nested conditional phrases may also be used to create complex queries. All nested phrases must also be enclosed with parenthesis. Allowed evaluators include: > >= < <= =
    Example (air_temp < 0) and (wind_speed > 10)

    Time Selection

    Argument Description Example
    start Provide a start and end date to only consider that time period for matching station observations. The time period cannot extend beyond the previous 48 hours from the current time. This value is in the form of YYYYMMDDHHmm as UTC. Only data in this period will be considered. When the period is finished being searched, the connection will automatically close.
    Example start=201801051800
    end Provide a start and end date to only consider that time period for matching station observations. The time period cannot extend beyond the previous 48 hours from the current time. This value is in the form of YYYYMMDDHHmm as UTC. Only data in this period will be considered. When the period is finished being searched, the connection will automatically close.
    Example end=201801052100
    rewind Provide a number of minutes behind the current time to start the feed. Once all observations have been returned for matches prior to the current time, the feed will resume in real-time mode. This option is useful to “seed” an application with recent data for trending.
    Example recent=5

    Units and Metadata

    Argument Description Example
    units Provide the units to return data in. Default is metric and the absence of this argument is metric. Note, the filter_on comparison values will match the units provided. For example, if mph is set for wind, then a filter of (wind_gust > 30) will look for gusts greater than 30 miles per hour.
    Example units=english or units=temp|f,wind|ms
    metadata Select if metadata is also returned within the responses. By default, all stations and units will return metadata. The metadata responses will always return within their own discrete messages in the feed, noted with the type of “metadata”. If metadata is not needed, adding a value of 0 or off will disable it. Metadata is returned exactly once for any unique station or unit seen within the feed. Metadata will always be sent prior to any observation the first time it is seen from any new connection. It is the developers responsibilty to organize metadata to be useful when applicable.