# Polars vs Tidyverse

<style>
pre, pre code, pre.sourceCode { font-size: 0.82em; line-height: 1.35; }
div.compare { display: flex; gap: 1.5rem; }
div.compare-col { flex: 1 1 0; min-width: 0; }
/* R text output (cell-output-stdout) gets a cream background by default
   from bootstrap's .cell-output.cell-output-stdout rule; override it so
   the tidyverse column matches the Polars (transparent) output. */
.cell-output.cell-output-stdout,
.cell-output.cell-output-stdout pre { background-color: transparent !important; }
@media screen {
  pre > code.sourceCode {
    white-space: pre-wrap;
    overflow-wrap: anywhere;
    word-break: break-word;
  }
  div.sourceCode { overflow-x: hidden; }
}
</style>


# Create the data


**Polars**


``` python
import polars as pl

adsl = pl.DataFrame({
    "USUBJID": ["U01", "U02", "U03", "U04"],
    "ARM":     ["DRUG A", "DRUG B", "DRUG A", "DRUG B"],
    "AGE":     [55, 62, 48, 70],
    "SEX":     ["M", "F", "M", "F"],
})

advs = pl.DataFrame({
    "USUBJID": ["U01","U01","U02","U02","U03","U03","U04","U04"],
    "ARM":     ["DRUG A","DRUG A","DRUG B","DRUG B","DRUG A","DRUG A","DRUG B","DRUG B"],
    "AGE":     [55, 55, 62, 62, 48, 48, 70, 70],
    "SEX":     ["M","M","F","F","M","M","F","F"],
    "PARAM":   ["Systolic BP","Diastolic BP","Systolic BP","Diastolic BP","Systolic BP","Diastolic BP","Systolic BP","Diastolic BP"],
    "PARAMCD": ["SYSBP","DIABP","SYSBP","DIABP","SYSBP","DIABP","SYSBP","DIABP"],
    "AVAL":    [120, 80, 130, 85, 110, 70, 140, 90],
    "AVISIT":  ["Visit 1","Visit 1","Visit 1","Visit 1","Visit 2","Visit 2","Visit 2","Visit 2"],
    "AVISITN": [1, 1, 1, 1, 2, 2, 2, 2],
    "DTC":     ["2024-01-15","2024-01-15","2024-02-20","2024-02-20","2024-03-10","2024-03-10","2024-04-05","2024-04-05"],
})
adsl
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
library(dplyr)
library(tidyr)
library(stringr)
library(lubridate)
library(readr)

adsl <- tibble::tibble(
  USUBJID = c("U01", "U02", "U03", "U04"),
  ARM     = c("DRUG A", "DRUG B", "DRUG A", "DRUG B"),
  AGE     = c(55, 62, 48, 70),
  SEX     = c("M", "F", "M", "F")
)

advs <- tibble::tibble(
  USUBJID = c("U01","U01","U02","U02","U03","U03","U04","U04"),
  ARM     = c("DRUG A","DRUG A","DRUG B","DRUG B","DRUG A","DRUG A","DRUG B","DRUG B"),
  AGE     = c(55, 55, 62, 62, 48, 48, 70, 70),
  SEX     = c("M","M","F","F","M","M","F","F"),
  PARAM   = c("Systolic BP","Diastolic BP","Systolic BP","Diastolic BP","Systolic BP","Diastolic BP","Systolic BP","Diastolic BP"),
  PARAMCD = c("SYSBP","DIABP","SYSBP","DIABP","SYSBP","DIABP","SYSBP","DIABP"),
  AVAL    = c(120, 80, 130, 85, 110, 70, 140, 90),
  AVISIT  = c("Visit 1","Visit 1","Visit 1","Visit 1","Visit 2","Visit 2","Visit 2","Visit 2"),
  AVISITN = c(1, 1, 1, 1, 2, 2, 2, 2),
  DTC     = c("2024-01-15","2024-01-15","2024-02-20","2024-02-20","2024-03-10","2024-03-10","2024-04-05","2024-04-05")
)
adsl
```


    # A tibble: 4 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U01     DRUG A    55 M    
    2 U02     DRUG B    62 F    
    3 U03     DRUG A    48 M    
    4 U04     DRUG B    70 F    


# Verbs


## Select columns


**Polars**


``` python
adsl.select("USUBJID", "ARM", "AGE")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> select(USUBJID, ARM, AGE)
```


    # A tibble: 4 × 3
      USUBJID ARM      AGE
      <chr>   <chr>  <dbl>
    1 U01     DRUG A    55
    2 U02     DRUG B    62
    3 U03     DRUG A    48
    4 U04     DRUG B    70


## Drop columns


**Polars**


``` python
adsl.drop("SEX")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> select(-SEX)
```


    # A tibble: 4 × 3
      USUBJID ARM      AGE
      <chr>   <chr>  <dbl>
    1 U01     DRUG A    55
    2 U02     DRUG B    62
    3 U03     DRUG A    48
    4 U04     DRUG B    70


## Drop by rule, not by name

R's `where(is.numeric)` and `starts_with()` have direct counterparts: `cs.numeric()` and `cs.starts_with()`, composed with `|` and `&`.


**Polars**


``` python
import polars.selectors as cs

adsl.select(cs.numeric() | cs.starts_with("U"))
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> select(where(is.numeric) | starts_with("U"))
```


    # A tibble: 4 × 2
        AGE USUBJID
      <dbl> <chr>  
    1    55 U01    
    2    62 U02    
    3    48 U03    
    4    70 U04    


## Rename columns

The dictionary is keyed old-to-new, the opposite of `rename()`'s `new = old`.


**Polars**


``` python
adsl.rename({"AGE": "AGE_N"})
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> rename(AGE_N = AGE)
```


    # A tibble: 4 × 4
      USUBJID ARM    AGE_N SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U01     DRUG A    55 M    
    2 U02     DRUG B    62 F    
    3 U03     DRUG A    48 M    
    4 U04     DRUG B    70 F    


## Filter rows

Conditions are expressions joined with `&`, `|`, `~`; `is_in` is `%in%`. Note that polars has no unquoted column names - every reference goes through `pl.col`.


**Polars**


``` python
advs.filter(
    (pl.col("ARM") == "DRUG A") & (pl.col("AVISIT").is_in(["Visit 1", "Visit 2"]))
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> filter(ARM == "DRUG A" & AVISIT %in% c("Visit 1", "Visit 2"))
```


    # A tibble: 4 × 10
      USUBJID ARM      AGE SEX   PARAM        PARAMCD  AVAL AVISIT  AVISITN DTC     
      <chr>   <chr>  <dbl> <chr> <chr>        <chr>   <dbl> <chr>     <dbl> <chr>   
    1 U01     DRUG A    55 M     Systolic BP  SYSBP     120 Visit 1       1 2024-01…
    2 U01     DRUG A    55 M     Diastolic BP DIABP      80 Visit 1       1 2024-01…
    3 U03     DRUG A    48 M     Systolic BP  SYSBP     110 Visit 2       2 2024-03…
    4 U03     DRUG A    48 M     Diastolic BP DIABP      70 Visit 2       2 2024-03…


## Add or replace columns


**Polars**


``` python
(
    advs
    .with_columns(doubled=pl.col("AVAL") * 2)
    .with_columns(squared=pl.col("doubled") ** 2)
    .select("USUBJID", "AVISIT", "AVAL", "doubled", "squared")
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  mutate(
    doubled = AVAL * 2,
    squared = doubled^2
  ) |>
  select(USUBJID, AVISIT, AVAL, doubled, squared)
```


    # A tibble: 8 × 5
      USUBJID AVISIT   AVAL doubled squared
      <chr>   <chr>   <dbl>   <dbl>   <dbl>
    1 U01     Visit 1   120     240   57600
    2 U01     Visit 1    80     160   25600
    3 U02     Visit 1   130     260   67600
    4 U02     Visit 1    85     170   28900
    5 U03     Visit 2   110     220   48400
    6 U03     Visit 2    70     140   19600
    7 U04     Visit 2   140     280   78400
    8 U04     Visit 2    90     180   32400


## Conditional values (case_when)


**Polars**


``` python
advs.with_columns(
    response=pl.when(pl.col("AVAL") > 120)
    .then(pl.lit("HIGH"))
    .otherwise(pl.lit("NORMAL"))
).select("USUBJID", "AVAL", "response")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  mutate(response = case_when(AVAL > 120 ~ "HIGH", TRUE ~ "NORMAL")) |>
  select(USUBJID, AVAL, response)
```


    # A tibble: 8 × 3
      USUBJID  AVAL response
      <chr>   <dbl> <chr>   
    1 U01       120 NORMAL  
    2 U01        80 NORMAL  
    3 U02       130 HIGH    
    4 U02        85 NORMAL  
    5 U03       110 NORMAL  
    6 U03        70 NORMAL  
    7 U04       140 HIGH    
    8 U04        90 NORMAL  


## Sort rows

The default placement of missing values differs: `arrange()` sorts `NA` last, `sort()` puts nulls first unless `nulls_last=True`.


**Polars**


``` python
adsl.sort("AGE", descending=True, nulls_last=True)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> arrange(desc(AGE))
```


    # A tibble: 4 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U04     DRUG B    70 F    
    2 U02     DRUG B    62 F    
    3 U01     DRUG A    55 M    
    4 U03     DRUG A    48 M    


## Slice by row position


**Polars**


``` python
adsl.head(3)
```

*\[Rich HTML output -- view on the documentation site\]*


``` python
adsl.tail(2)
```

*\[Rich HTML output -- view on the documentation site\]*


``` python
adsl.slice(1, 2)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> slice_head(n = 3)
```


    # A tibble: 3 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U01     DRUG A    55 M    
    2 U02     DRUG B    62 F    
    3 U03     DRUG A    48 M    


``` r
adsl |> slice_tail(n = 2)
```


    # A tibble: 2 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U03     DRUG A    48 M    
    2 U04     DRUG B    70 F    


``` r
adsl |> slice(2:3)
```


    # A tibble: 2 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U02     DRUG B    62 F    
    2 U03     DRUG A    48 M    


## Distinct


**Polars**


``` python
advs.select("USUBJID", "ARM").unique().sort("USUBJID")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> distinct(USUBJID, ARM) |> arrange(USUBJID)
```


    # A tibble: 4 × 2
      USUBJID ARM   
      <chr>   <chr> 
    1 U01     DRUG A
    2 U02     DRUG B
    3 U03     DRUG A
    4 U04     DRUG B


## Type conversion


**Polars**


``` python
adsl.with_columns(id=pl.col("USUBJID").str.extract(r"(\d+)").cast(pl.Int32)).select("USUBJID", "id")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> mutate(id = as.integer(str_extract(USUBJID, "\\d+"))) |> select(USUBJID, id)
```


    # A tibble: 4 × 2
      USUBJID    id
      <chr>   <int>
    1 U01         1
    2 U02         2
    3 U03         3
    4 U04         4


## Batch rename (prefix / suffix)


**Polars**


``` python
adsl.select(pl.col("AGE").name.prefix("BASE_"))
```

*\[Rich HTML output -- view on the documentation site\]*


``` python
adsl.select(pl.all().name.suffix("_raw"))
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
adsl |> select(AGE) |> rename_with(~ str_c("BASE_", .x))
```


    # A tibble: 4 × 1
      BASE_AGE
         <dbl>
    1       55
    2       62
    3       48
    4       70


``` r
adsl |> rename_with(~ str_c(.x, "_raw"))
```


    # A tibble: 4 × 4
      USUBJID_raw ARM_raw AGE_raw SEX_raw
      <chr>       <chr>     <dbl> <chr>  
    1 U01         DRUG A       55 M      
    2 U02         DRUG B       62 F      
    3 U03         DRUG A       48 M      
    4 U04         DRUG B       70 F      


# Missing values


## Check for missing values


**Polars**


``` python
advs.null_count()
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> summarise(across(everything(), ~ sum(is.na(.))))
```


    # A tibble: 1 × 10
      USUBJID   ARM   AGE   SEX PARAM PARAMCD  AVAL AVISIT AVISITN   DTC
        <int> <int> <int> <int> <int>   <int> <int>  <int>   <int> <int>
    1       0     0     0     0     0       0     0      0       0     0


## Drop rows with a missing value


**Polars**


``` python
miss = advs.with_columns(AVAL=pl.when(pl.col("USUBJID") == "U02").then(None).otherwise(pl.col("AVAL")))
miss.drop_nulls("AVAL").select("USUBJID", "AVAL")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
miss <- advs |> mutate(AVAL = ifelse(USUBJID == "U02", NA, AVAL))
miss |> drop_na(AVAL) |> select(USUBJID, AVAL)
```


    # A tibble: 6 × 2
      USUBJID  AVAL
      <chr>   <dbl>
    1 U01       120
    2 U01        80
    3 U03       110
    4 U03        70
    5 U04       140
    6 U04        90


## Fill with a constant


**Polars**


``` python
miss.with_columns(filled=pl.col("AVAL").fill_null(0)).select("USUBJID", "AVAL", "filled")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
miss |>
  mutate(filled = replace_na(AVAL, 0)) |>
  select(USUBJID, AVAL, filled)
```


    # A tibble: 8 × 3
      USUBJID  AVAL filled
      <chr>   <dbl>  <dbl>
    1 U01       120    120
    2 U01        80     80
    3 U02        NA      0
    4 U02        NA      0
    5 U03       110    110
    6 U03        70     70
    7 U04       140    140
    8 U04        90     90


## Carry the last value forward


**Polars**


``` python
miss.with_columns(
    filled=pl.col("AVAL").fill_null(strategy="forward")
).select("USUBJID", "AVISIT", "AVAL", "filled")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
miss |>
  fill(AVAL, .direction = "down") |>
  select(USUBJID, AVISIT, AVAL)
```


    # A tibble: 8 × 3
      USUBJID AVISIT   AVAL
      <chr>   <chr>   <dbl>
    1 U01     Visit 1   120
    2 U01     Visit 1    80
    3 U02     Visit 1    80
    4 U02     Visit 1    80
    5 U03     Visit 2   110
    6 U03     Visit 2    70
    7 U04     Visit 2   140
    8 U04     Visit 2    90


## Fill within a group


**Polars**


``` python
miss.with_columns(
    filled=pl.col("AVAL").fill_null(strategy="forward").over("USUBJID")
).select("USUBJID", "AVISIT", "AVAL", "filled")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
miss |>
  group_by(USUBJID) |>
  fill(AVAL, .direction = "down") |>
  ungroup() |>
  select(USUBJID, AVISIT, AVAL)
```


    # A tibble: 8 × 3
      USUBJID AVISIT   AVAL
      <chr>   <chr>   <dbl>
    1 U01     Visit 1   120
    2 U01     Visit 1    80
    3 U02     Visit 1    NA
    4 U02     Visit 1    NA
    5 U03     Visit 2   110
    6 U03     Visit 2    70
    7 U04     Visit 2   140
    8 U04     Visit 2    90


## Coalesce

`pl.coalesce` takes columns, `coalesce()` takes vectors.


**Polars**


``` python
co = pl.DataFrame({"a": [1, None, 3], "b": [10, 20, None]})
co.with_columns(c=pl.coalesce("a", "b"))
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
co <- tibble(a = c(1, NA, 3), b = c(10, 20, NA))
co |> mutate(c = coalesce(a, b))
```


    # A tibble: 3 × 3
          a     b     c
      <dbl> <dbl> <dbl>
    1     1    10     1
    2    NA    20    20
    3     3    NA     3


## Delete empty rows


**Polars**


``` python
blank = pl.DataFrame({"USUBJID": [None], "AVAL": [None]})
pl.concat([miss.select("USUBJID", "AVAL"), blank], how="vertical").drop_nulls()
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
blank <- tibble(USUBJID = NA_character_, AVAL = NA_real_)
bind_rows(miss |> select(USUBJID, AVAL), blank) |> drop_na()
```


    # A tibble: 6 × 2
      USUBJID  AVAL
      <chr>   <dbl>
    1 U01       120
    2 U01        80
    3 U03       110
    4 U03        70
    5 U04       140
    6 U04        90


# Group and window


## Summarise by group


**Polars**


``` python
advs.group_by("ARM").agg(
    n=pl.len(),
    mean_age=pl.col("AGE").mean(),
    mean_aval=pl.col("AVAL").mean(),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(ARM) |>
  summarise(
    n = n(),
    mean_age = mean(AGE),
    mean_aval = mean(AVAL)
  )
```


    # A tibble: 2 × 4
      ARM        n mean_age mean_aval
      <chr>  <int>    <dbl>     <dbl>
    1 DRUG A     4     51.5       95 
    2 DRUG B     4     66        111.


## Common statistics


**Polars**


``` python
advs.select(
    mean=pl.col("AVAL").mean(),
    median=pl.col("AVAL").median(),
    sd=pl.col("AVAL").std(),
    p75=pl.col("AVAL").quantile(0.75),
    total=pl.col("AVAL").sum(),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  summarise(
    mean = mean(AVAL),
    median = median(AVAL),
    sd = sd(AVAL),
    p75 = quantile(AVAL, 0.75),
    total = sum(AVAL)
  )
```


    # A tibble: 1 × 5
       mean median    sd   p75 total
      <dbl>  <dbl> <dbl> <dbl> <dbl>
    1  103.    100  25.5  122.   825


## Summarise many columns at once (across)


**Polars**


``` python
advs.group_by("ARM").agg(
    pl.col("AGE", "AVAL").mean().name.prefix("mean_"),
    pl.col("AGE", "AVAL").std().name.prefix("sd_"),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(ARM) |>
  summarise(across(c(AGE, AVAL), list(mean = mean, sd = sd)))
```


    # A tibble: 2 × 5
      ARM    AGE_mean AGE_sd AVAL_mean AVAL_sd
      <chr>     <dbl>  <dbl>     <dbl>   <dbl>
    1 DRUG A     51.5   4.04       95     23.8
    2 DRUG B     66     4.62      111.    27.8


## Count rows per group

`len()` is `n()`.


**Polars**


``` python
advs.group_by("ARM").len()
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> count(ARM)
```


    # A tibble: 2 × 2
      ARM        n
      <chr>  <int>
    1 DRUG A     4
    2 DRUG B     4


## Window value (recycle an aggregate)


**Polars**


``` python
advs.with_columns(
    subj_mean=pl.col("AVAL").mean().over("USUBJID")
).select("USUBJID", "AVISIT", "AVAL", "subj_mean")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(USUBJID) |>
  mutate(subj_mean = mean(AVAL)) |>
  ungroup() |>
  select(USUBJID, AVISIT, AVAL, subj_mean)
```


    # A tibble: 8 × 4
      USUBJID AVISIT   AVAL subj_mean
      <chr>   <chr>   <dbl>     <dbl>
    1 U01     Visit 1   120      100 
    2 U01     Visit 1    80      100 
    3 U02     Visit 1   130      108.
    4 U02     Visit 1    85      108.
    5 U03     Visit 2   110       90 
    6 U03     Visit 2    70       90 
    7 U04     Visit 2   140      115 
    8 U04     Visit 2    90      115 


## Running total within a group


**Polars**


``` python
advs.with_columns(
    cum_aval=pl.col("AVAL").cum_sum().over("USUBJID")
).select("USUBJID", "AVISIT", "AVAL", "cum_aval")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(USUBJID) |>
  mutate(cum_aval = cumsum(AVAL)) |>
  ungroup() |>
  select(USUBJID, AVISIT, AVAL, cum_aval)
```


    # A tibble: 8 × 4
      USUBJID AVISIT   AVAL cum_aval
      <chr>   <chr>   <dbl>    <dbl>
    1 U01     Visit 1   120      120
    2 U01     Visit 1    80      200
    3 U02     Visit 1   130      130
    4 U02     Visit 1    85      215
    5 U03     Visit 2   110      110
    6 U03     Visit 2    70      180
    7 U04     Visit 2   140      140
    8 U04     Visit 2    90      230


## Lag and lead


**Polars**


``` python
advs.with_columns(
    prev=pl.col("AVAL").shift(1).over("USUBJID"),
    lead_val=pl.col("AVAL").shift(-1).over("USUBJID"),
).select("USUBJID", "AVISIT", "AVAL", "prev", "lead_val")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(USUBJID) |>
  mutate(
    prev = lag(AVAL),
    lead_val = lead(AVAL)
  ) |>
  ungroup() |>
  select(USUBJID, AVISIT, AVAL, prev, lead_val)
```


    # A tibble: 8 × 5
      USUBJID AVISIT   AVAL  prev lead_val
      <chr>   <chr>   <dbl> <dbl>    <dbl>
    1 U01     Visit 1   120    NA       80
    2 U01     Visit 1    80   120       NA
    3 U02     Visit 1   130    NA       85
    4 U02     Visit 1    85   130       NA
    5 U03     Visit 2   110    NA       70
    6 U03     Visit 2    70   110       NA
    7 U04     Visit 2   140    NA       90
    8 U04     Visit 2    90   140       NA


## First and last row of each group


**Polars**


``` python
(
    advs
    .with_columns(
        rn=pl.col("AVAL").rank(method="ordinal").over("USUBJID", "AVISITN"),
        is_first=pl.col("AVAL").rank(method="ordinal").over("USUBJID", "AVISITN") == 1,
        is_last=pl.col("AVAL").rank(method="ordinal").over("USUBJID", "AVISITN") == pl.len().over("USUBJID", "AVISITN"),
    )
    .select("USUBJID", "AVISIT", "rn", "is_first", "is_last")
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  group_by(USUBJID, AVISITN) |>
  mutate(
    rn = row_number(),
    is_first = row_number() == 1,
    is_last = row_number() == n()
  ) |>
  ungroup() |>
  select(USUBJID, AVISIT, rn, is_first, is_last)
```


    # A tibble: 8 × 5
      USUBJID AVISIT     rn is_first is_last
      <chr>   <chr>   <int> <lgl>    <lgl>  
    1 U01     Visit 1     1 TRUE     FALSE  
    2 U01     Visit 1     2 FALSE    TRUE   
    3 U02     Visit 1     1 TRUE     FALSE  
    4 U02     Visit 1     2 FALSE    TRUE   
    5 U03     Visit 2     1 TRUE     FALSE  
    6 U03     Visit 2     2 FALSE    TRUE   
    7 U04     Visit 2     1 TRUE     FALSE  
    8 U04     Visit 2     2 FALSE    TRUE   


## Row-wise sum of several columns


**Polars**


``` python
advs.with_columns(total=pl.sum_horizontal("AGE", "AVAL")).select("USUBJID", "AGE", "AVAL", "total")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  mutate(total = AGE + AVAL) |>
  select(USUBJID, AGE, AVAL, total)
```


    # A tibble: 8 × 4
      USUBJID   AGE  AVAL total
      <chr>   <dbl> <dbl> <dbl>
    1 U01        55   120   175
    2 U01        55    80   135
    3 U02        62   130   192
    4 U02        62    85   147
    5 U03        48   110   158
    6 U03        48    70   118
    7 U04        70   140   210
    8 U04        70    90   160


# Join and reshape


## Left join


**Polars**


``` python
arms = pl.DataFrame({"ARM": ["DRUG A", "DRUG B"], "ARMCD": ["A", "B"]})
adsl.join(arms, on="ARM", how="left")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
arms <- tibble(ARM = c("DRUG A", "DRUG B"), ARMCD = c("A", "B"))
adsl |> left_join(arms, by = "ARM")
```


    # A tibble: 4 × 5
      USUBJID ARM      AGE SEX   ARMCD
      <chr>   <chr>  <dbl> <chr> <chr>
    1 U01     DRUG A    55 M     A    
    2 U02     DRUG B    62 F     B    
    3 U03     DRUG A    48 M     A    
    4 U04     DRUG B    70 F     B    


## Set operations


**Polars**


``` python
a = pl.DataFrame({"USUBJID": ["U01", "U02", "U03"]})
b = pl.DataFrame({"USUBJID": ["U02", "U03", "U04"]})
a.filter(pl.col("USUBJID").is_in(b["USUBJID"].to_list()))
```

*\[Rich HTML output -- view on the documentation site\]*


``` python
a.join(b, on="USUBJID", how="anti")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
a <- tibble(USUBJID = c("U01", "U02", "U03"))
b <- tibble(USUBJID = c("U02", "U03", "U04"))
a |> filter(USUBJID %in% b$USUBJID)
```


    # A tibble: 2 × 1
      USUBJID
      <chr>  
    1 U02    
    2 U03    


``` r
a |> anti_join(b, by = "USUBJID")
```


    # A tibble: 1 × 1
      USUBJID
      <chr>  
    1 U01    


## Pivot long to wide


**Polars**


``` python
wide = advs.pivot(index="USUBJID", on="PARAMCD", values="AVAL")
wide
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
wide <- advs |> pivot_wider(names_from = PARAMCD, values_from = AVAL)
wide
```


    # A tibble: 8 × 10
      USUBJID ARM      AGE SEX   PARAM        AVISIT  AVISITN DTC        SYSBP DIABP
      <chr>   <chr>  <dbl> <chr> <chr>        <chr>     <dbl> <chr>      <dbl> <dbl>
    1 U01     DRUG A    55 M     Systolic BP  Visit 1       1 2024-01-15   120    NA
    2 U01     DRUG A    55 M     Diastolic BP Visit 1       1 2024-01-15    NA    80
    3 U02     DRUG B    62 F     Systolic BP  Visit 1       1 2024-02-20   130    NA
    4 U02     DRUG B    62 F     Diastolic BP Visit 1       1 2024-02-20    NA    85
    5 U03     DRUG A    48 M     Systolic BP  Visit 2       2 2024-03-10   110    NA
    6 U03     DRUG A    48 M     Diastolic BP Visit 2       2 2024-03-10    NA    70
    7 U04     DRUG B    70 F     Systolic BP  Visit 2       2 2024-04-05   140    NA
    8 U04     DRUG B    70 F     Diastolic BP Visit 2       2 2024-04-05    NA    90


## Pivot wide to long


**Polars**


``` python
wide.unpivot(index="USUBJID", on=["SYSBP","DIABP"], variable_name="PARAMCD", value_name="AVAL")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
wide |> pivot_longer(c(SYSBP, DIABP), names_to = "PARAMCD", values_to = "AVAL")
```


    # A tibble: 16 × 10
       USUBJID ARM      AGE SEX   PARAM        AVISIT  AVISITN DTC     PARAMCD  AVAL
       <chr>   <chr>  <dbl> <chr> <chr>        <chr>     <dbl> <chr>   <chr>   <dbl>
     1 U01     DRUG A    55 M     Systolic BP  Visit 1       1 2024-0… SYSBP     120
     2 U01     DRUG A    55 M     Systolic BP  Visit 1       1 2024-0… DIABP      NA
     3 U01     DRUG A    55 M     Diastolic BP Visit 1       1 2024-0… SYSBP      NA
     4 U01     DRUG A    55 M     Diastolic BP Visit 1       1 2024-0… DIABP      80
     5 U02     DRUG B    62 F     Systolic BP  Visit 1       1 2024-0… SYSBP     130
     6 U02     DRUG B    62 F     Systolic BP  Visit 1       1 2024-0… DIABP      NA
     7 U02     DRUG B    62 F     Diastolic BP Visit 1       1 2024-0… SYSBP      NA
     8 U02     DRUG B    62 F     Diastolic BP Visit 1       1 2024-0… DIABP      85
     9 U03     DRUG A    48 M     Systolic BP  Visit 2       2 2024-0… SYSBP     110
    10 U03     DRUG A    48 M     Systolic BP  Visit 2       2 2024-0… DIABP      NA
    11 U03     DRUG A    48 M     Diastolic BP Visit 2       2 2024-0… SYSBP      NA
    12 U03     DRUG A    48 M     Diastolic BP Visit 2       2 2024-0… DIABP      70
    13 U04     DRUG B    70 F     Systolic BP  Visit 2       2 2024-0… SYSBP     140
    14 U04     DRUG B    70 F     Systolic BP  Visit 2       2 2024-0… DIABP      NA
    15 U04     DRUG B    70 F     Diastolic BP Visit 2       2 2024-0… SYSBP      NA
    16 U04     DRUG B    70 F     Diastolic BP Visit 2       2 2024-0… DIABP      90


## Stack frames vertically

`concat([...], how="vertical")` is `bind_rows`.


**Polars**


``` python
top = advs.head(2)
bot = advs.tail(2)
pl.concat([top, bot], how="vertical")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
top <- advs |> head(2)
bot <- advs |> tail(2)
bind_rows(top, bot)
```


    # A tibble: 4 × 10
      USUBJID ARM      AGE SEX   PARAM        PARAMCD  AVAL AVISIT  AVISITN DTC     
      <chr>   <chr>  <dbl> <chr> <chr>        <chr>   <dbl> <chr>     <dbl> <chr>   
    1 U01     DRUG A    55 M     Systolic BP  SYSBP     120 Visit 1       1 2024-01…
    2 U01     DRUG A    55 M     Diastolic BP DIABP      80 Visit 1       1 2024-01…
    3 U04     DRUG B    70 F     Systolic BP  SYSBP     140 Visit 2       2 2024-04…
    4 U04     DRUG B    70 F     Diastolic BP DIABP      90 Visit 2       2 2024-04…


## Stack frames side by side


**Polars**


``` python
left = adsl.select("USUBJID", "ARM")
right = adsl.select("AGE", "SEX")
pl.concat([left, right], how="horizontal_extend")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
left <- adsl |> select(USUBJID, ARM)
right <- adsl |> select(AGE, SEX)
bind_cols(left, right)
```


    # A tibble: 4 × 4
      USUBJID ARM      AGE SEX  
      <chr>   <chr>  <dbl> <chr>
    1 U01     DRUG A    55 M    
    2 U02     DRUG B    62 F    
    3 U03     DRUG A    48 M    
    4 U04     DRUG B    70 F    


# Strings


## Change case


**Polars**


``` python
advs.select(
    upper=pl.col("ARM").str.to_uppercase(),
    lower=pl.col("PARAMCD").str.to_lowercase(),
    title=pl.col("PARAM").str.to_titlecase(),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> transmute(
  upper = str_to_upper(ARM),
  lower = str_to_lower(PARAMCD),
  title = str_to_title(PARAM)
)
```


    # A tibble: 8 × 3
      upper  lower title       
      <chr>  <chr> <chr>       
    1 DRUG A sysbp Systolic Bp 
    2 DRUG A diabp Diastolic Bp
    3 DRUG B sysbp Systolic Bp 
    4 DRUG B diabp Diastolic Bp
    5 DRUG A sysbp Systolic Bp 
    6 DRUG A diabp Diastolic Bp
    7 DRUG B sysbp Systolic Bp 
    8 DRUG B diabp Diastolic Bp


## Trim and measure length


**Polars**


``` python
advs.select(
    n=pl.col("PARAM").str.len_chars(),
    trimmed=pl.col("PARAM").str.strip_chars(),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> transmute(
  n = str_length(PARAM),
  trimmed = str_trim(PARAM)
)
```


    # A tibble: 8 × 2
          n trimmed     
      <int> <chr>       
    1    11 Systolic BP 
    2    12 Diastolic BP
    3    11 Systolic BP 
    4    12 Diastolic BP
    5    11 Systolic BP 
    6    12 Diastolic BP
    7    11 Systolic BP 
    8    12 Diastolic BP


## Test for a pattern


**Polars**


``` python
advs.filter(pl.col("PARAMCD").str.contains(r"^S"))
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> filter(str_detect(PARAMCD, "^S"))
```


    # A tibble: 4 × 10
      USUBJID ARM      AGE SEX   PARAM       PARAMCD  AVAL AVISIT  AVISITN DTC      
      <chr>   <chr>  <dbl> <chr> <chr>       <chr>   <dbl> <chr>     <dbl> <chr>    
    1 U01     DRUG A    55 M     Systolic BP SYSBP     120 Visit 1       1 2024-01-…
    2 U02     DRUG B    62 F     Systolic BP SYSBP     130 Visit 1       1 2024-02-…
    3 U03     DRUG A    48 M     Systolic BP SYSBP     110 Visit 2       2 2024-03-…
    4 U04     DRUG B    70 F     Systolic BP SYSBP     140 Visit 2       2 2024-04-…


## Extract the first match


**Polars**


``` python
advs.with_columns(num=pl.col("USUBJID").str.extract(r"(\d+)")).select("USUBJID", "num")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> mutate(num = str_extract(USUBJID, "\\d+")) |> select(USUBJID, num)
```


    # A tibble: 8 × 2
      USUBJID num  
      <chr>   <chr>
    1 U01     01   
    2 U01     01   
    3 U02     02   
    4 U02     02   
    5 U03     03   
    6 U03     03   
    7 U04     04   
    8 U04     04   


## Replace a pattern


**Polars**


``` python
advs.with_columns(arm=pl.col("ARM").str.replace(r"DRUG ", "")).select("ARM", "arm")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> mutate(arm = str_remove(ARM, "DRUG ")) |> select(ARM, arm)
```


    # A tibble: 8 × 2
      ARM    arm  
      <chr>  <chr>
    1 DRUG A A    
    2 DRUG A A    
    3 DRUG B B    
    4 DRUG B B    
    5 DRUG A A    
    6 DRUG A A    
    7 DRUG B B    
    8 DRUG B B    


## Split and merge


**Polars**


``` python
advs.select(
    parts=pl.col("PARAM").str.split(" "),
    merged=pl.col("USUBJID") + pl.lit("-") + pl.col("AVISIT"),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> transmute(
  parts = str_split(PARAM, " "),
  merged = str_c(USUBJID, "-", AVISIT)
)
```


    # A tibble: 8 × 2
      parts     merged     
      <list>    <chr>      
    1 <chr [2]> U01-Visit 1
    2 <chr [2]> U01-Visit 1
    3 <chr [2]> U02-Visit 1
    4 <chr [2]> U02-Visit 1
    5 <chr [2]> U03-Visit 2
    6 <chr [2]> U03-Visit 2
    7 <chr [2]> U04-Visit 2
    8 <chr [2]> U04-Visit 2


## Slice and pad


**Polars**


``` python
advs.select(
    sliced=pl.col("PARAMCD").str.slice(0, 3),
    padded=pl.col("USUBJID").str.zfill(6),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> transmute(
  sliced = str_sub(PARAMCD, 1, 3),
  padded = str_pad(USUBJID, width = 6, side = "left", pad = "0")
)
```


    # A tibble: 8 × 2
      sliced padded
      <chr>  <chr> 
    1 SYS    000U01
    2 DIA    000U01
    3 SYS    000U02
    4 DIA    000U02
    5 SYS    000U03
    6 DIA    000U03
    7 SYS    000U04
    8 DIA    000U04


# Dates


## Parse a date, datetime or time


**Polars**


``` python
advs.with_columns(
    date=pl.col("DTC").str.to_date(),
).select("USUBJID", "date")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> mutate(date = ymd(DTC)) |> select(USUBJID, date)
```


    # A tibble: 8 × 2
      USUBJID date      
      <chr>   <date>    
    1 U01     2024-01-15
    2 U01     2024-01-15
    3 U02     2024-02-20
    4 U02     2024-02-20
    5 U03     2024-03-10
    6 U03     2024-03-10
    7 U04     2024-04-05
    8 U04     2024-04-05


## Pull out parts


**Polars**


``` python
advs.with_columns(
    d=pl.col("DTC").str.to_date(),
).select(
    year=pl.col("d").dt.year(),
    month=pl.col("d").dt.month(),
    day=pl.col("d").dt.day(),
    wday=pl.col("d").dt.weekday(),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  mutate(d = ymd(DTC)) |>
  transmute(
    year = year(d),
    month = month(d),
    day = mday(d),
    wday = wday(d)
  )
```


    # A tibble: 8 × 4
       year month   day  wday
      <dbl> <dbl> <int> <dbl>
    1  2024     1    15     2
    2  2024     1    15     2
    3  2024     2    20     3
    4  2024     2    20     3
    5  2024     3    10     1
    6  2024     3    10     1
    7  2024     4     5     6
    8  2024     4     5     6


## Format a date as text


**Polars**


``` python
advs.with_columns(
    txt=pl.col("DTC").str.to_date().dt.strftime("%d %b %Y")
).select("DTC", "txt")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> mutate(txt = format(ymd(DTC), "%d %b %Y")) |> select(DTC, txt)
```


    # A tibble: 8 × 2
      DTC        txt        
      <chr>      <chr>      
    1 2024-01-15 15 Jan 2024
    2 2024-01-15 15 Jan 2024
    3 2024-02-20 20 Feb 2024
    4 2024-02-20 20 Feb 2024
    5 2024-03-10 10 Mar 2024
    6 2024-03-10 10 Mar 2024
    7 2024-04-05 05 Apr 2024
    8 2024-04-05 05 Apr 2024


## Truncate to a period


**Polars**


``` python
advs.with_columns(
    month_start=pl.col("DTC").str.to_date().dt.truncate("1mo"),
).select("DTC", "month_start")
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |> mutate(month_start = floor_date(ymd(DTC), "month")) |> select(DTC, month_start)
```


    # A tibble: 8 × 2
      DTC        month_start
      <chr>      <date>     
    1 2024-01-15 2024-01-01 
    2 2024-01-15 2024-01-01 
    3 2024-02-20 2024-02-01 
    4 2024-02-20 2024-02-01 
    5 2024-03-10 2024-03-01 
    6 2024-03-10 2024-03-01 
    7 2024-04-05 2024-04-01 
    8 2024-04-05 2024-04-01 


## Month start and end


**Polars**


``` python
(
    advs
    .with_columns(d=pl.col("DTC").str.to_date())
    .with_columns(
        ms=pl.col("d").dt.month_start(),
        me=pl.col("d").dt.month_end(),
    )
    .select("DTC", "ms", "me")
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
advs |>
  mutate(
    d = ymd(DTC),
    ms = floor_date(d, "month"),
    me = ceiling_date(d, "month") - days(1)
  ) |>
  select(DTC, ms, me)
```


    # A tibble: 8 × 3
      DTC        ms         me        
      <chr>      <date>     <date>    
    1 2024-01-15 2024-01-01 2024-01-31
    2 2024-01-15 2024-01-01 2024-01-31
    3 2024-02-20 2024-02-01 2024-02-29
    4 2024-02-20 2024-02-01 2024-02-29
    5 2024-03-10 2024-03-01 2024-03-31
    6 2024-03-10 2024-03-01 2024-03-31
    7 2024-04-05 2024-04-01 2024-04-30
    8 2024-04-05 2024-04-01 2024-04-30


## Add or subtract a period


**Polars**


``` python
jan = pl.DataFrame({"d": ["2024-01-31"]}).with_columns(d=pl.col("d").str.to_date())
jan.with_columns(
    next_month=pl.col("d").dt.offset_by("1mo"),
    prev_month=pl.col("d").dt.offset_by("-1mo"),
)
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
jan <- tibble(d = ymd("2024-01-31"))
jan |>
  mutate(
    next_month = d %m+% months(1),
    prev_month = d %m-% months(1)
  )
```


    # A tibble: 1 × 3
      d          next_month prev_month
      <date>     <date>     <date>    
    1 2024-01-31 2024-02-29 2023-12-31


## Difference between dates


**Polars**


``` python
jan = pl.DataFrame({"d1": ["2024-01-01"], "d2": ["2024-04-01"]}).with_columns(
    pl.col("d1", "d2").str.to_date()
)
jan.with_columns(days=pl.col("d2") - pl.col("d1"))
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
jan <- tibble(d1 = ymd("2024-01-01"), d2 = ymd("2024-04-01"))
jan |> mutate(days = d2 - d1)
```


    # A tibble: 1 × 3
      d1         d2         days   
      <date>     <date>     <drtn> 
    1 2024-01-01 2024-04-01 91 days


## Generate a sequence of dates


**Polars**


``` python
pl.DataFrame({"d": pl.date_range(pl.date(2024, 1, 1), pl.date(2024, 1, 5), "1d", eager=True)})
```

*\[Rich HTML output -- view on the documentation site\]*


**tidyverse**


``` r
tibble(d = seq(ymd("2024-01-01"), ymd("2024-01-05"), by = "day"))
```


    # A tibble: 5 × 1
      d         
      <date>    
    1 2024-01-01
    2 2024-01-02
    3 2024-01-03
    4 2024-01-04
    5 2024-01-05


# Input / output


## Read and write CSV


**Polars**


``` python
# write
advs.write_csv("advs.csv")
# read
advs = pl.read_csv("advs.csv", try_parse_dates=True)
```


**tidyverse**


``` r
# write
advs |> write_csv("advs.csv")
# read
advs <- read_csv("advs.csv")
```


## Read and write Parquet


**Polars**


``` python
advs.write_parquet("advs.parquet")
advs = pl.read_parquet("advs.parquet")
```


**tidyverse**


``` r
advs |> arrow::write_parquet("advs.parquet")
advs <- arrow::read_parquet("advs.parquet")
```


# Sources

- Polars Python API. <https://docs.pola.rs/api/python/stable/reference/>.
- Wickham H, Averick M, Bryan J, et al. Welcome to the tidyverse. *Journal of Open Source Software*, 2019;4(43):1686. <https://doi.org/10.21105/joss.01686>.
- Grolemund G, Wickham H. Dates and Times Made Easy with lubridate. *Journal of Statistical Software*, 2011;40(3):1-25. <https://doi.org/10.18637/jss.v040.i03>.