Apache Parquet Writer Parameters

Dataset

The compression type to apply to each column when writing the file. Use this option to compress and reduce file size.

  • UNCOMPRESSED – This is the default.
  • ZSTD – This option provides a good compression ratio across diverse datasets. ZSTD works well with CHAR and VARCHAR columns that store a wide range of long and short strings, including JSON strings.
  • Snappy – This option provides a reasonable compression ratio, but fast compression speeds.

Determines use of Parquet Format version types. The option is defaulted to write version 2.0.

Note  Some external Parquet implementations may not support types added in the 2.0.0 format version. To maximize compatibility of your files, you may want to change the parameter to 1.0.

When this option group is enabled, the writer will write a partitioned dataset instead of a single .parquet file in the writer's specified output directory. See the Parquet File Extensions section for more details.

Overwrite Existing Dataset

Check this option to overwrite an existing partitioned dataset.

Warning  After a dataset is deleted, it cannot be recovered – all data is lost.
  • If this option is unchecked (default) and a partitioned dataset already exists, the writer will error. Features cannot be written to an existing dataset and this avoids unexpectedly deleting data.
  • If this option is checked, any existing directories or files in the partitioned dataset directory will be deleted before writing begins.

If a dataset does not exist when writing begins, a new dataset is created.

Partition Type

  • Hive – When partitioning by an attribute, the subdirectory will be named in the form attrName=attrValue. This way of partitioning was introduced by Apache Hive.
    • Note that attribute values partitioned via the Hive partition type will be URI-encoded. This means that special characters (like spaces, question marks, ampersands, hashes, parentheses, braces, brackets, and punctuation) will be encoded.
    • For example, an attribute called Name with value John Smith would result in a partitioned subdirectory called Name=John%20Smith.
  • Directory – When partitioning by an attribute, the subdirectory will be named in the form attrValue. This is a simple type of directory partitioning.

    • Note that attribute values partitioned via the Directory partition type containing forward or backward slashes will result in additional subdirectories.

    • For example, an attribute called Date with value 2023/01/02 would result in a partitioned Parquet file under nested directories named 2023, 01, and 02.