Parquet Access

ONETICK MARKET DATA | OPEN SOURCE WITH NO VENDOR LOCK-IN

 

Parquet is an open-source, column-oriented data file format created for highly efficient data storage and retrieval.

Data can be retrieved and queried using many query tools, removing vendor lock-in.

OneTick delivers tick by tick historic market data in Parquet format to optimize big data analytics

Parquet is a highly efficient, columnar storage file format that drastically reduces storage costs and improves read speeds.

Using Parquet files allows quantitative researchers and data scientists to stream and parse complex, high-frequency datasets—such as Level 1, Level 2, and Level 3 order book data—directly into cloud storage or local research environments.

Loading

OneTick loads Parquet from raw exchange PCAP files across Equites, Futures and Options, into standardized schemas. OneTick also loads the schema that each exchanges provides, ensuring relevant information from PCAPs are not lost, while adding standardized fields to support cross venue analysis.

OneTick creates AI Feature sets from the raw trade, quotes and book depth. These are also provided in parquet format.

The Parquet files allow you to easily query decades of historical microstructure, trades, and order lifecycles at petabyte scale without prohibitive infrastructure overhead.

Storage

All historic market data is stored in parquet, and made available on the AWS s3 bucket.

s3://otc-shared-parquet-storage

Data is stored in the s3 bucket under the following categories:

  • Consolidated

  • Derivatives

  • Equity

  • Fx

  • Index

  • Sample

Venues and Consolidated Regions are stored under each category. Tables are grouped into data granularity (tick, bar and daily). Each Table is represented as a set of partitioned parquet, partitioned by Date.

AWS Tools

Customers with Parquet access can download, and directly query data using AWS tools.

AWS credentials, specifically aws_access_key_id and aws_secret_access_key are provided to subscribing customers.

Examples are provided showing how to:

  1. Use the AWS CLI to list, download and synchronize files.

  2. Use the AWS Boto3 Python library to list and download files in Python.

  3. Use the PyArrow library to directly querying the Parquet in Python

Analytics

Parquet does not provide an analytics layer. You can use your own analytics platform to query the Parquet, or additionally leverage OneTick Cloud Analytics for faster analytics execution, and faster analytics development.