Pipelines

The real power of rush is combining commands in Unix pipelines. Because rush reads from stdin and writes to stdout by default, it composes naturally with other command-line tools, including other rush invocations.

Chaining rush commands

Filter with rush run, then query with SQL:

rush run 'dplyr::filter(df, body_mass_g > 3500)' penguins.csv | \
  rush sql "SELECT species, ROUND(AVG(bill_length_mm), 1) as avg_bill FROM stdin GROUP BY species ORDER BY avg_bill DESC" -
#> species,avg_bill
#> Chinstrap,49.7
#> Gentoo,47.6
#> Adelie,39.7

Query with SQL, then filter further with rush run:

rush sql "SELECT species, bill_length_mm, body_mass_g FROM penguins WHERE body_mass_g > 5000" penguins.csv | \
  rush run 'head(df, 5)' -
#> species,bill_length_mm,body_mass_g
#> Gentoo,50,5700
#> Gentoo,50,5700
#> Gentoo,47.6,5400
#> Gentoo,46.7,5200
#> Gentoo,46.8,5150

JSON in pipelines

Produce JSON, process it, and convert back to CSV:

rush run -O json 'head(df, 3)' penguins.csv | rush run -F json 'dplyr::select(df, species, body_mass_g)' -
#> species,body_mass_g
#> Adelie,3750
#> Adelie,3800
#> Adelie,3250

Format conversion in pipelines

Read Parquet, filter with SQL, output as JSON:

rush sql -O json "SELECT carrier, origin, dest FROM flights WHERE dep_delay > 20" flights.parquet
#> [
#>   {
#>     "carrier": "AA",
#>     "origin": "JFK",
#>     "dest": "LAX"
#>   },
#>   {
#>     "carrier": "DL",
#>     "origin": "LGA",
#>     "dest": "ATL"
#>   },
#>   {
#>     "carrier": "WN",
#>     "origin": "EWR",
#>     "dest": "DEN"
#>   }
#> ]

Multiple rush commands

Chain rush run to progressively transform data:

rush run 'dplyr::select(df, species, bill_length_mm, body_mass_g)' penguins.csv | \
  rush run 'dplyr::filter(df, body_mass_g > 5000)' -
#> species,bill_length_mm,body_mass_g
#> Gentoo,50,5700
#> Gentoo,50,5700
#> Gentoo,47.6,5400
#> Gentoo,46.7,5200
#> Gentoo,46.8,5150
#> Gentoo,49,5550
#> Gentoo,48.4,5850
#> Gentoo,49.3,5850
#> Gentoo,49.2,6300
#> Gentoo,48.7,5350
#> Gentoo,50.2,5700
#> Gentoo,46.3,5050
#> Gentoo,46.1,5100
#> Gentoo,47.8,5650
#> Gentoo,50,5550
#> Gentoo,47.3,5250
#> Gentoo,45.1,5050
#> Gentoo,59.6,6050
#> Gentoo,49.1,5150
#> Gentoo,48.4,5400
#> Gentoo,44.4,5250
#> Gentoo,48.7,5350
#> Gentoo,49.6,5700
#> Gentoo,50.5,5550
#> Gentoo,50.5,5400
#> Gentoo,44.9,5100
#> Gentoo,45.2,5300
#> Gentoo,48.5,5300
#> Gentoo,45,5050
#> Gentoo,50.4,5550
#> Gentoo,46.2,5300
#> Gentoo,54.3,5650
#> Gentoo,49.8,5700
#> Gentoo,49.5,5800
#> Gentoo,50.7,5550
#> Gentoo,48.2,5100
#> Gentoo,46.5,5200
#> Gentoo,48.6,5800
#> Gentoo,51.1,6000
#> Gentoo,45.2,5950
#> Gentoo,52.5,5450
#> Gentoo,50,5350
#> Gentoo,50.8,5600
#> Gentoo,51.3,5300
#> Gentoo,52.1,5550
#> Gentoo,52.2,5400
#> Gentoo,49.5,5650
#> Gentoo,50.8,5200
#> Gentoo,51.1,5250
#> Gentoo,55.9,5600
#> Gentoo,49.1,5500
#> Gentoo,46.8,5500
#> Gentoo,53.4,5500
#> Gentoo,48.1,5500
#> Gentoo,49.8,5950
#> Gentoo,51.5,5500
#> Gentoo,55.1,5850
#> Gentoo,48.8,6000
#> Gentoo,50.4,5750
#> Gentoo,45.2,5200
#> Gentoo,49.9,5400

With other tools

rush plays well with standard Unix tools:

rush run 'df' penguins.csv | wc -l
#> 334

rush run 'df' penguins.csv | cut -d, -f1 | sort | uniq -c
#>     146 Adelie
#>      68 Chinstrap
#>     119 Gentoo
#>       1 species

Feed external data into rush:

curl -s https://api.example.com/data.json | rush run -F json 'head(df)'