Push Streaming

Synoptic Push Streaming Services

The data push services allow users to “plug” into the real-time data streams after data checks have been applied. Nearly 100 million observations flow through this system every day, and these data services allow for the filtering of various domains and observed surface conditions from over 260 mesonet networks and over 120,000 active stations.

Data streaming is available for customers with our higher tier commercial accounts.

Connection and Data Format Details

Establishing Connections

The connection protocol uses secure websockets (wss://). Establishing a connection to the data push services requires a wss connection to our server endpoint. The connection arguments can take one of two forms: initial connections and reestablishing connections.

Initial Connections

NONE

wss://push.synopticdata.com/feed/{valid_api_token}/?{optional_arguments}

This form expects a valid API token and any optional arguments to limit the selection domain. For example:

wss://push.synopticdata.com/feed/{valid_api_token}/?state=ca&vars=air_temp,wind_speed

In this case, all observations originating from stations within the state of California that report air temperature or wind speed will send to the established connection within seconds of the observation entering our data ingest system.

The initial response will take this form

JSON

{
    "code": "success",
    "messages": ["Starting new session."],
    "session": "993624a1-2563-412e-9cfb-647535a364ac",
    "type": "auth"
}

or

JSON

{
    "code": "failed",
    "messages": ["invalid token"],
    "session": "",
    "type": "auth"
}

Re-establishing Connections

wss://push.synopticdata.com/feed/[your token]/993624a1-2563-412e-9cfb-647535a364ac

Upon successful initial connections, a session ID is provided that can be used to reestablish a severed or stopped connection. This is useful to ensure all data since the last push is resumed at the exact location without missing a single observation. The use of this form will ignore all optional arguments and use the arguments supplied with the initial connection if any were given.

The response for reestablishing connections will take this form

JSON

{
    "code": "success",
    "messages": ["Restarting from provided session id."],
    "session": "33258f22-77b7-4514-8a30-a792e97e170a",
    "type": "auth"
}

Response Types

All response objects returned from the data push service are valid JSON. Every JSON object will contain the key of “type” which indicates exactly what the object represents. There are 3 types.

JSON Reponse Type	Description
`auth`	Used to indicate the status of attempted authentication. Successful authentications will provide a session ID.
`metadata`	Used to indicate all information in the JSON object is metadata relating to UNITS or STATIONS.
`data`	Used to indicate all information in the JSON object is observational data matching the domain criteria provided for the active session.

Stopping Connections

Connections can be explicitly stopped at any time. Sending the command to terminate the existing websocket connection takes this form

JSON

{
    "feed_action": "stop"
}

Stopping a connection or simply severing the network connection is effectively the same thing. If the connection is reestablished, all data not pushed since the last successful push will begin to flow.

Data Object Response Formats

By default, basic metadata of all data elements are also sent. Metadata will always flow prior to the first occurrence of data, which guarantees metadata is available for any given observational data object. The metadata format takes this form

JSON

{
    "units": [
        {
            "sensor": "air_temp",
            "unit": "Celsius"
        },
        {
            "sensor": "wind_speed",
            "unit": "m/s"
        }
    ],
    "type": "metadata",
    "stations": [
        {
            "latitude": "40.76623",
            "stid": "WBB",
            "elevation": 4806.0,
            "network": 153,
            "longitude": "-111.84755"
        },
        {
            "latitude": "38.89724",
            "stid": "MOA18",
            "elevation": 829.0,
            "network": 156,
            "longitude": "-92.21807"
        },
        ...
    ]
}

Objects containing observational data will have this form

JSON

{       
    "type": "data",
    "data": [
        {
            "set": 1,
            "stid": "KXBP",
            "value": -6.69,
            "qc": [],
            "date": 201801132215,
            "sensor": "wind_speed"
        },
        {
            "set": 1,
            "stid": "C4824",
            "value": 6.56,
            "qc": [],
            "date": 201801132220,
            "sensor": "air_temp"
        },
        {
            "set": 1,
            "stid": "C4824",
            "value": 0.0,
            "qc": [],
            "date": 201801132220,
            "sensor": "wind_speed"
        },
        {
            "set": 1,
            "stid": "D7524",
            "value": -8.33,
            "qc": [],
            "date": 201801132221,
            "sensor": "air_temp"
        }
    ]
}

The “data” entry is a list of observations broken into individual variable values. This pattern follows other data services such as the Weather API. Each nested object of data contain the following keys:

Key	Description
`stid`	The station identifier known within all Synoptic data services.
`date`	The date and time of the observation. This is in the form of yyyymmddhhmm (year, month, day, hour, minute) in UTC.
`sensor`	The sensor (variable) name the data represents. The names used are the same as all Synoptic data services (see API).
`set`	The instance of the sensor for the given station. Some stations may report more than one instance of a sensor, such as air temperature at 2 meters and 10 meters, or soil temperature at 10 cm below the surface and 30 cm below the surface.
`value`	The value of the sensed observation for that moment in time. The units will be in what was supplied on the initial connection. Default is metric. The metadata JSON object will contain all unit information for clarification.
`qc`	A list of all data checks and quality control flags that indicate a failed test. For example, `sl_rate_change` reflects a rate change test failed for the given observation.

Web Socket Connection Arguments

Domain Selection

Argument	Description Example
`state`	Provide a list of state codes to limit the observation domain to consider. Example `state=ut,ca,nv`
`network`	Provide a list of network IDs to consider. All other observations are ignored. Example `network=1,65,2,153`
`radius`	Provide a latitude and longitude with a radius in miles. Only observations originating from the radius will be considered. Alternatively, provide a station ID and a radius. Example `radius=41.5,-111.2,50 or radius=KLAX,100`
`nwszone`	Provide a list of NWS county warning areas as the domain to consider. Example `nwszone=VA525,UT003`
`nwsfirezone`	Provide a list of NWS fire zones as the domain to consider. Example `nwsfirezone=SLC478`
`bbox`	Provide a bounding box as the domain to listen to. The rectangle must be in the form of (`min_lon`, `min_lat`, `max_lon`, `max_lat`). Example `bbox=-100,30,-90,40`
`stid`	Provide a distinct set of stations to monitor. Only these stations will have data returned. Example `stid=KSLC,WBB,KSFO`

Sensor and Data Checks

Argument

Description Example

vars

Provide a list of sensor names to limit returning only those data elements. Sensor names are in the form found on the Weather API. All names are identical. The response returned will contain these names as the sensor as well. Omitting this argument will return all available sesnsors for the matching stations within the selection domain.
Example vars=air_temp,wind_speed,wind_gust,pressure,relative_humidity

qc

A list of data checks that remove observations matching the given list. By default, no observations will be returned that have been flagged with a failed range test (sl_range_check). It is up to the user to decide what flags are important to apply to the data based on the use case.
Example qc=sl_rate_change,sl_persistence

Filter Conditions

Argument

Description Example

filter_on

Set of conditional expressions defined similar to a SQL where statement. Only data matching the criteria set will be returned in the feed. Each condition must be enclosed within parenthesis and have a sensor name, evaluator, and value. Conditional phrases may be chained together with “and” or “or” operators. Nested conditional phrases may also be used to create complex queries. All nested phrases must also be enclosed with parenthesis. Allowed evaluators include: > >= < <= =
Example (air_temp < 0) and (wind_speed > 10)

Time Selection

Argument	Description Example
`rewind`	Provide a number of minutes behind the current time to start the feed. Once all observations have been returned for matches prior to the current time, the feed will resume in real-time mode. This option is useful to “seed” an application with recent data for trending. Example `rewind=5`

Units and Metadata

Argument

Description Example

units

Provide the units to return data in. Default is metric and the absence of this argument is metric. Note, the filter_on comparison values will match the units provided. For example, if mph is set for wind, then a filter of (wind_gust > 30) will look for gusts greater than 30 miles per hour.
Example units=english or units=temp|f,wind|ms

metadata

Select if metadata is also returned within the responses. By default, all stations and units will return metadata. The metadata responses will always return within their own discrete messages in the feed, noted with the type of “metadata”. If metadata is not needed, adding a value of 0 or off will disable it. Metadata is returned exactly once for any unique station or unit seen within the feed. Metadata will always be sent prior to any observation the first time it is seen from any new connection. It is the developers responsibilty to organize metadata to be useful when applicable.