rewriting history

What this document is and What is not:

This document describes procedure to upload historical data so it became available through the historical API, in other words "re-writing history"

Standard flow for writing historical data:

The standard procedure to populate the HISTORICAL DATA component is through the Context Manager, as described in this diagram:


+------------+     1      +----------+      3       +---------------+
|            |POST /entity|          | POST /notify |               |
|            | +--------> |  Context | +----------> |    Historical |
|    DCA     |     ok     |  Manager |    ok        |    Data       |
|            | <----------+          | <---------+  |               |
+------------+            +----------+              +---------------+
                    2                       4

                             (orion)                   (quantumLeap)

entity data encapsulated in flow (1) is real time data, IoT platform assumes this data happened "just now", hence this flow is not adequate to write history.

Why not?

If we used this flow and we tried to write 1000 records in a series of POST request, we be modifying first the current value many times, and hence be pushing to the HISTORICAL DATA a 1000 records to dated as today :/
Also, it is not safe to circulate great volumes of data through the Context Manager, it was not built for this purpose.

What is documented here is how to achieve this, while at the same time indicating the exact data this info took place :

`rewriting-history` flow for writing historical data:

                      3
 +------------+  POST /notify +---------------+
 |            |  +----------> |               |
 |            |     ok        |    Historical |
 |    DCA     |  <---------+  |    Data       |
 |            |               |               |
 +------------+       4       +---------------+


                                 (quantumLeap)

PROCEDURE

Preconditions:

[ ] You need to have cURL or some way of querying with an HTTP API

[ ] Data Model of the entity to be pushed in the HISTORICAL DATA needs to be STABLE

[ ] You need to have data formatted using the right Data Model

STEP 1 - Get the entity looks from the IoT platform

This for making sure that entity which are trying to rewrite history exists in CONTEXT MANAGER and follows a correct data model.

Dont forget 'Fiware-Service` in the http headers!

curl --location --request GET 'http://5.53.108.182:1026/v2/entities/urn:ngsi-ld:WeatherObserved:WeatherObserved-1?metadata=dateCreated,dateModified' \
--header 'Fiware-Service: alicante'

{
    "id": "urn:ngsi-ld:WeatherObserved:WeatherObserved-1",
    "type": "WeatherObserved",
    "atmosphericPressure": {
        "type": "Number",
        "value": 0,
        "metadata": {
            "dateCreated": {
                "type": "DateTime",
                "value": "2020-10-20T13:45:57.00Z"
            },
            "dateModified": {
                "type": "DateTime",
                "value": "2020-10-20T13:45:57.00Z"
            }
        }
    },
    "temperature": {
        "type": "Number",
        "value": 23.2,
        "metadata": {
            "dateCreated": {
                "type": "DateTime",
                "value": "2020-10-20T13:45:57.00Z"
            },
            "dateModified": {
                "type": "DateTime",
                "value": "2020-10-21T08:24:47.00Z"
            }
        }
    },
    "dewPoint": {
        "type": "Number",
        "value": 13.2,
        "metadata": {
            "dateCreated": {
                "type": "DateTime",
                "value": "2020-10-20T13:45:57.00Z"
            },
            "dateModified": {
                "type": "DateTime",
                "value": "2020-10-20T13:52:43.00Z"
            }
        }
(...)

IMPORTANT copy and save the result somewhere, this data model is the one that needs used for pushing data to the HISTORICAL DATA component

STEP 2 - Check standard flow for writing history data is already in place

Before pushing the data to the Verify HISTORICAL DATA is already exposing data for this entity, e.g.:

dont forget the headers!

curl --location --request GET 'http://5.53.108.182:8668/v2/entities/urn:ngsi-ld:WeatherObserved:WeatherObserved-1/attrs/temperature/value?lastN=10' \
--header 'Fiware-Service: alicante' \
--header 'Fiware-ServicePath: /'

#this returns:

{
    "index": [
        "2020-10-20T13:52:43.000",
        "2020-10-21T08:24:47.000"
    ],
    "values": [
        0.0,
        23.2
    ]
}

If this step fails then contact UDGA for subscribing your entity to the HISTORICAL DATA API

STEP 3 - Get your historical data subscription id

For pushing data we will need something called "subscription id" of the entity withing the HISTORICAL DATA API, here a list of the existing ones:

for carouge:
5f071e73e44ddaac278abd3d: FlowerBed
5f60dcf399fc2ae29f141b92: WaterQualityObserved
5f747649da7e7c9ac9757a6c: Device
5f8f021eda7e7c9ac9757a76: WeatherObserved

for braila:
5f8f0229da7e7c9ac9757a78: WeatherObserved 

for alicante:
5f8f0225da7e7c9ac9757a77: WeatherObserved

If the entity you want to push is not listed here then it means that this entity is not being saved at all in the HISTORICAL DATA.

If this step fails then contact UDGA for subscribing your entity to the HISTORICAL DATA API

STEP 4 - `rewrite-history`

Now we are ready to write history! Let's start by writing a single registry into the HISTORICAL DATA

Here below and example on how to push data to Alicante's urn:ngsi-ld:WeatherObserved:WeatherObserved-1 entity, here we are only going to send an update to temperature attribute of the entity.

Note well:

sending a partial description of the entity like this will make all the other attributes to take value "Null" , e.g wind attribute, so it's better to push the complete entity rather than just one value (unless all the other attributes dont really matter for historical purposes)
you need to include for each attribute the dateModified metadata, else current datetime is going to be used, this is:

"metadata": {
   "dateModified": {
     "type": "DateTime",
     "value": "2020-10-21T16:16:16.00Z"
   }
}

Here below an example on how to push a single record using cURL:

curl --location --request POST 'http://5.53.108.182:8668/v2/notify' \
--header 'Fiware-Service: alicante' \
--header 'Fiware-ServicePath: /' \
--header 'Content-Type: application/json' \
--data-raw '{ 
    "subscriptionId": "5f8f0225da7e7c9ac9757a77", 
    "data": [ 
        {
            "id": "urn:ngsi-ld:WeatherObserved:WeatherObserved-1",
            "type": "WeatherObserved",
            "temperature": {
                "type": "Number",
                "value": 1000,
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T16:16:16.00Z"
                    }
                }
            }
        }
    ] 
}
'

STEP 5 - Automate and post all your historical data

Now we are ready to push all our data. Here you will have to get your hands dirty, meaning that you will need to implement in some way automation code for pushing all those entities one by one using the API.

If you know how to use python, then may contact UDGA and ask for an example for doing this.