Parquet

Parquet is a file type that has gained wide spread adoption due to it’s efficiency. Parquet files can be read and written much faster than .fits files. They are also internally organized more efficiently, meaning they take up less disk space and less memory when in use. Due to it’s efficiency, SDSS began releasing parquet versions of some large files in DR20.

Popular software for interacting with Parquet files includes Polars and PyArrow. pandas can also be used directly, with multiple parquet backends available including PyArrow. We can demonstrate the efficiency of parquet files quickly, in this case with a fairly small file (162MB on disk as fits, 32 MB as parquet).

In [1]: import sys
   ...: from time import time
   ...: import pandas as pd
   ...: import fitsio

In [2]: t1 = time()
   ...: parquet = pd.read_parquet("parquet/mos_sdssv_boss_spall.parquet")
   ...: print(time() - t1)
1.2226293087005615

In [3]: t2 = time()
   ...: fits = fitsio.read("fits/mos_sdssv_boss_spall.fits")
   ...: print(time() - t2)
3.4632937908172607

In [4]: print(f"Parquet: {sys.getsizeof(parquet)/1e6:.2f} MB")
   ...: print(f"FITS: {sys.getsizeof(fits)/1e6:.2f} MB")
Parquet: 164.19 MB
FITS: 412.11 MB

Efficiencies in reading files vary; while this example is roughly a 3x improvement, improvements of more than 10x are common for larger files. The memory efficiency is obvious, and also becomes more substantial for larger files.

Using pandas to read the parquet file yields a typical pandas dataframe that can be interacted with by specifying columns:

In [5]: parquet["sn_median_all"]
Out[5]: 
0          3.590165
1          7.767848
2         31.443550
3          3.188130
4          3.076941
            ...    
195495     0.496741
195496     0.202830
195497    47.178623
195498    34.809181
195499    32.444828
Name: sn_median_all, Length: 195500, dtype: float32

Polars provides tools for interacting with large files without reading the entire file into memory, a so called “lazy API”. This feature is not exclusive to parquet files, but it is well optimized for parquet files. For DR20, two tutorials are provided to demonstrate usage of SDSS V targeting files. The tutorial for parquet files leans heavily on polars. The tutorial for fits files notes that it is only recommended to run on machines with a full copy of the SAS (e.g. SciServer) because the file volume is so large.

Back to Top