Target Mapping

Targets supporting a database column structure map fields from Listener to specific columns. All targets, except HDFS types, support mapping fields. You can map data to columns using passthrough or automatic mapping.

Passthrough Mapping

Listener wraps messages with the following metadata you can map and store in the target:

FieldDescription
dataField with the raw data sent to Ingest.
source_idSource ID to which the packet was sent.
timeTime the packet was ingested.
uuidUUID gets assigned to the packet upon arrival.

You can link any of the fields described above to any of the columns described below:

ColumnType
recordsjson
streamvarchar
timestamptimestamp
idvarchar
The data_map.mapping array requires an object with three properties – column, field, and type – where column is the Teradata database column and field is what appears on the Listener user interface. The following is an example of defining target mapping.
{
  "data_map": {
    "mapping": [
      {
        "column": "customer_data",
        "field": "data",
        "type": "json"
      }, {
        "column": "customer_source_id",
        "field": "source_id",
        "type": "varchar"
      }, {
        "column": "timestamp",
        "field": "time",
        "type": "timestamp"
      }, {
        "column": "id",
        "field": "uuid",
        "type": "varchar"
      }
    ]
  }
}

Automatic Mapping

Automatic mapping binds JSON values to specific data types in Teradata Database columns. Listener parses each message as a JSON document and maps the parsed fields to Teradata Database columns.

The following data types are supported:
Data TypeDescription
BigintListener 2.00.02
Binary (Teradata Byte)Listener 2.00.02
BlobListener 2.00.02
CharListener 2.00.02
DateListener 2.00.02
Supported format:
  • YYYY-MM-DD
DecimalListener 2.00.02
DoubleListener 2.00.02
FloatListener 2.00.02
IntegerListener 2.00.02
LongvarbinaryListener 2.00.02
SmallintListener 2.00.02
TimestampListener 2.00.02
Supported format:
  • YYYY-MM-DD hh:mi:ss.ssssss
TinyintListener 2.00.02
VarbinaryListener 2.00.02
VarcharListener 2.00, 2.00.01, 2.00.02

Automatic mapping is exposed in Listener through the data_map.mapping_type using auto_shred. The writer parses the JSON and makes the following fields and values available for binding to the target database table.

{
  "data_map": {
    "mapping_type" : "auto_shred"
  }
}
The following is an example of how data is written when a target is configured with auto_shred and you provide valid JSON:
{
  "country" : "France",
  "population" : 66991000,
  "famous_for" : ["Wine", "Cheese"],
  "restaurants" : {
    "French" : 63,
    "Italian" : 25,
    "Vegan" : 9,
    "Other" : 3,
  },
  "monarch" : null
}
FieldTypeContents
countryParsed"France"
populationParsed66991000
famous_forParsed"["Wine","Cheese"]"
restaurantsParsed"{"French": 63, "Italian": 25, "Vegan": 9, "Other": 3}"
monarchParsednull
Each table column in the database corresponds with a JSON field, as shown in the following example:
Table ColumnValueDescription
COUNTRYFranceMatches country field through case insensitive comparison.
UUIDUUID associated with packetMatches uuid field through case insensitive comparison.
TIMEZONEnullNo value; defaults to null.

results matching ""

    No results matching ""