Data Pools
Data Pools are the raw data storage for the system. Data Pools can be constructed to hold any tabular data that is needed to fuel subsequently created Data Views.
At a basic level Data Pools can be thought of as relational database tables, with some caveats:
- No relational joins. Data Pools are designed for high performance analytics. As a result, they do not support data normalized between pools. As much as possible, data should be frozen in place, pre-computed , and isolated; to allow for fast column aggregations.
- No definable primary key. Primary keys such as auto incrementing fields can not be created. A simple field can be created to hold the primary key for an external dataset, but constraint enforcement or incrementing cant be configured.
- Not a source of truth. Data Pools are not intended to be a source of truth. Generally, a Data Pool will be populated from some external dataset and used for computing analytical queries. Although, data lime ephemeral logs and events work as well.
Data types
Data Pool fields are typed. Each field can only store data that corresponds to its designated data type. The following data types are available:
| Type | Description |
|---|---|
| Int8 | An integer between -128 and 127 |
| Int16 | An integer between -32768 and 32767 |
| Int32 | An integer between -2147483648 and 2147483647 |
| Int64 | An integer between -9223372036854775808 and 9223372036854775807 |
| UInt8 | An integer between 0 and 255 |
| UInt16 | An integer between 0 and 65535 |
| UInt32 | An integer between 0 and 4294967295 |
| UInt64 | An integer between 0 and 18446744073709551615 |
| Float32 | A 32-bit floating point number |
| Float64 | A 64-bit floating point number |
| String | A String of characters of arbitrary length |
| Boolean | A true or false value |
| Date | A date without time information |
| DateTime | A date and time value, expected to be in UTC |
| UUID | A 128-bit UUID represented as 32 hex characters |
| Array | An array of values of a given data type |
Creating Data Pools
- From within an organization, click
Data Poolsin the side menu. - Click the
Create Datapoolbutton, which will open the Data Pool creation model. - Give the Data Pool a name and click the
Createbutton.
Creating fields
For a Data Pool to be functional it needs fields. To add fields to a Data Pool select the Fields tab and click Create Field.
In the create field modal, give the field a name and a data type. the name will be used when making ingestion requests. The data type sets the type of data that can be stored in the field.
Deployment
When a Data Pool is first created, it is in an undeployed state. In this state it is unusable and consumes no resources. On the details page for a, recently created, Data Pool clicking Deploy will create the Data Pool.
Once deployed, the Data Pool is in an inactive state. While inactive the Data Pool is provisioned and maintains its data, but will not allow new ingestion, or data egress through views.
To active the Data Pool click Activate. In an active state, data can be ingested into the Data Pool from the ingestion endpoint. Data can also be retrieved from the Data Pool by views.