Retrieve the list of Parquet files of the analytical table and their stored checksums.

To do this, execute the following command on the server with the metadata catalog:

  SELECT df.data_file_id, df.path, df.checksum_crc32
  FROM axe_catalog.pga_data_file df
  JOIN axe_catalog.pga_table t ON df.table_id = t.table_id
  WHERE t.table_name = 'table_name'
  AND df.end_snapshot IS NULL;

Where table_name is the name of the analytical table containing the Parquet file that must be checked.

With a tool that uses the CRC-32-IEEE 802.3 algorithm, calculate the actual checksum of the Parquet file that must be checked.

  from zlib import crc32
  path = '/home/user/local_storage_example/file.parquet'
  with open(path, 'rb') as fp:
      data = fp.read()
      print(crc32(data))

Compare the computed checksum of the Parquet file with the checksum stored in the checksum_crc32 column of the pga_data_file metadata table.

If the values match, the file is intact. Otherwise, the file is corrupted.