Parquet

Files ending in .parquet or .pq are read with nanoparquet, no flags needed. Point --output (-o) at a .parquet file to write one.

Reading

rush run 'df |> dplyr::filter(dep_delay > 10) |> dplyr::select(carrier, origin, dest, dep_delay)' flights.parquet
#> carrier,origin,dest,dep_delay
#> DL,LGA,ATL,12
#> AA,JFK,LAX,120
#> DL,LGA,ATL,22
#> WN,EWR,DEN,45

Writing

Convert CSV to Parquet and read it back:

rush run --output penguins.parquet 'df' penguins.csv
rush run 'head(df, 3)' penguins.parquet
#> species,island,bill_length_mm,bill_depth_mm,flipper_length_mm,body_mass_g,sex,year
#> Adelie,Torgersen,39.1,18.7,181,3750,male,2007
#> Adelie,Torgersen,39.5,17.4,186,3800,female,2007
#> Adelie,Torgersen,40.3,18,195,3250,female,2007

With SQL

Because DuckDB reads Parquet natively, large-file queries run without loading everything into memory:

rush sql "SELECT carrier, COUNT(*) AS flights, ROUND(AVG(arr_delay), 1) AS avg_delay
          FROM flights
          GROUP BY carrier
          ORDER BY avg_delay DESC" flights.parquet
#> carrier,flights,avg_delay
#> WN,1,50
#> AA,3,33.7
#> DL,3,7
#> UA,3,-4.3