# Getting started

Datasets are module attributes, lazy-loaded from parquet.


# Collections

| Module                       | Standard | Source          | Datasets |
|------------------------------|----------|-----------------|---------:|
| `pharmadata.pharmaverseadam` | ADaM     | pharmaverseadam |       30 |
| `pharmadata.pharmaversesdtm` | SDTM     | pharmaversesdtm |       64 |
| `pharmadata.cdiscpilotadam`  | ADaM     | CDISC pilot     |       12 |
| `pharmadata.cdiscpilotsdtm`  | SDTM     | CDISC pilot     |       31 |


# Install

``` bash
pip install pharmadata
```


# Usage


``` python
import polars as pl
from pharmadata import pharmaverseadam as adam, pharmaversesdtm as sdtm
```


``` python
adam.list_datasets()
adam.meta_tables()
adam.meta_columns()
adam.meta_specs()
adam_dict = adam.to_dict()
```


``` python
adsl = adam.adsl
adsl_meta = adam.adsl_meta
adsl_meta.column_names
adsl_meta.column_labels
adsl_meta.column_names_to_labels
adsl_meta.number_rows
adsl_meta.number_columns
adsl_meta.file_label
adsl_meta.file_encoding
adsl.head()
```


shape: (5, 55)

| STUDYID | USUBJID | SUBJID | SITEID | COUNTRY | RFSTDTC | RFENDTC | RFXSTDTC | RFXENDTC | RFPENDTC | SCRFDT | FRVDT | DTHDTC | DTHADY | DTHFL | LDDTHELD | LDDTHGR1 | DTH30FL | DTHA30FL | DTHDOM | DTHB30FL | REGION1 | DMDTC | DMDY | AGE | AGEU | AGEGR1 | SEX | RACE | RACEGR1 | ETHNIC | SAFFL | ARM | ARMCD | ACTARM | ACTARMCD | TRT01P | TRT01A | TRTSDT | TRTSDTM | TRTSTMF | TRTEDT | TRTEDTM | TRTETMF | EOSSTT | EOSDT | RFICDTC | RANDDT | LSTALVDT | TRTDURD | DTHDT | DTHDTF | DTHCAUS | DTHCGR1 | BRTHDTC |
|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|
| str | str | str | str | str | str | str | str | str | str | date | date | str | f64 | str | f64 | str | str | str | str | str | str | str | f64 | f64 | str | str | str | str | str | str | str | str | str | str | str | str | str | date | datetime\[μs, UTC\] | str | date | datetime\[μs, UTC\] | str | str | date | str | date | date | f64 | date | str | str | str | str |
| "CDISCPILOT01" | "01-701-1015" | "1015" | "701" | "USA" | "2014-01-02" | "2014-07-02" | "2014-01-02" | "2014-07-02" | "2014-07-02T11:45" | null | null | null | null | null | null | null | null | null | null | null | "NA" | "2013-12-26" | -7.0 | 63.0 | "YEARS" | "18-64" | "F" | "WHITE" | "White" | "HISPANIC OR LATINO" | "Y" | "Placebo" | "Pbo" | "Placebo" | "Pbo" | "Placebo" | "Placebo" | 2014-01-02 | 2014-01-02 00:00:00 UTC | "H" | 2014-07-02 | 2014-07-02 23:59:59 UTC | "H" | "COMPLETED" | 2014-07-02 | null | 2014-01-02 | 2014-07-02 | 182.0 | null | null | null | null | "1950-12-26" |
| "CDISCPILOT01" | "01-701-1023" | "1023" | "701" | "USA" | "2012-08-05" | "2012-09-02" | "2012-08-05" | "2012-09-01" | "2013-02-18" | null | 2013-02-18 | null | null | null | null | null | null | null | null | null | "NA" | "2012-07-22" | -14.0 | 64.0 | "YEARS" | "18-64" | "M" | "WHITE" | "White" | "HISPANIC OR LATINO" | "Y" | "Placebo" | "Pbo" | "Placebo" | "Pbo" | "Placebo" | "Placebo" | 2012-08-05 | 2012-08-05 00:00:00 UTC | "H" | 2012-09-01 | 2012-09-01 23:59:59 UTC | "H" | "DISCONTINUED" | 2012-09-02 | null | 2012-08-05 | 2012-09-02 | 28.0 | null | null | null | null | "1948-07-22" |
| "CDISCPILOT01" | "01-701-1028" | "1028" | "701" | "USA" | "2013-07-19" | "2014-01-14" | "2013-07-19" | "2014-01-14" | "2014-01-14T11:10" | null | null | null | null | null | null | null | null | null | null | null | "NA" | "2013-07-11" | -8.0 | 71.0 | "YEARS" | "\>64" | "M" | "WHITE" | "White" | "NOT HISPANIC OR LATINO" | "Y" | "Xanomeline High Dose" | "Xan_Hi" | "Xanomeline High Dose" | "Xan_Hi" | "Xanomeline High Dose" | "Xanomeline High Dose" | 2013-07-19 | 2013-07-19 00:00:00 UTC | "H" | 2014-01-14 | 2014-01-14 23:59:59 UTC | "H" | "COMPLETED" | 2014-01-14 | null | 2013-07-19 | 2014-01-14 | 180.0 | null | null | null | null | "1942-07-11" |
| "CDISCPILOT01" | "01-701-1033" | "1033" | "701" | "USA" | "2014-03-18" | "2014-04-14" | "2014-03-18" | "2014-03-31" | "2014-09-15" | null | 2014-09-15 | null | null | null | null | null | null | null | null | null | "NA" | "2014-03-10" | -8.0 | 74.0 | "YEARS" | "\>64" | "M" | "WHITE" | "White" | "NOT HISPANIC OR LATINO" | "Y" | "Xanomeline Low Dose" | "Xan_Lo" | "Xanomeline Low Dose" | "Xan_Lo" | "Xanomeline Low Dose" | "Xanomeline Low Dose" | 2014-03-18 | 2014-03-18 00:00:00 UTC | "H" | 2014-03-31 | 2014-03-31 23:59:59 UTC | "H" | "DISCONTINUED" | 2014-04-14 | null | 2014-03-18 | 2014-04-14 | 14.0 | null | null | null | null | "1940-03-10" |
| "CDISCPILOT01" | "01-701-1034" | "1034" | "701" | "USA" | "2014-07-01" | "2014-12-30" | "2014-07-01" | "2014-12-30" | "2014-12-30T09:50" | null | null | null | null | null | null | null | null | null | null | null | "NA" | "2014-06-24" | -7.0 | 77.0 | "YEARS" | "\>64" | "F" | "WHITE" | "White" | "NOT HISPANIC OR LATINO" | "Y" | "Xanomeline High Dose" | "Xan_Hi" | "Xanomeline High Dose" | "Xan_Hi" | "Xanomeline High Dose" | "Xanomeline High Dose" | 2014-07-01 | 2014-07-01 00:00:00 UTC | "H" | 2014-12-30 | 2014-12-30 23:59:59 UTC | "H" | "COMPLETED" | 2014-12-30 | null | 2014-07-01 | 2014-12-30 | 183.0 | null | null | null | null | "1937-06-24" |


# Format


``` python
from pharmadata import set_output_pandas, set_output_polars, output_format
set_output_pandas()
dm = sdtm.dm
print(type(dm))
set_output_polars()
dm = sdtm.dm
print(type(dm))
```


    <class 'pandas.DataFrame'>
    <class 'polars.dataframe.frame.DataFrame'>


[set_output_pandas()](../reference/set_output_pandas.md#pharmadata.set_output_pandas) / [set_output_polars()](../reference/set_output_polars.md#pharmadata.set_output_polars) switch globally; [output_format()](../reference/output_format.md#pharmadata.output_format) is a context manager.


``` python
with output_format("pandas"):
    adsl_pd = adam.adsl

print(type(adsl_pd))
```


    <class 'pandas.DataFrame'>


# Validate by `pointblank`


``` python
import pointblank as pb
# pb.validate_adam(adsl, "ADSL")
pb.validate_adam_structure(adsl, "ADSL")
```


    {'dataset': 'ADSL',
     'dataset_class': 'ADSL',
     'valid': True,
     'missing_required': [],
     'missing_conditional': [],
     'unknown_variables': [],
     'population_flags_found': ['SAFFL'],
     'issues': []}


# Polars Examples


``` python
adsl.select(["STUDYID", "USUBJID", "AGE", "SEX", "TRT01A"]).head()
```


shape: (5, 5)

| STUDYID        | USUBJID       | AGE  | SEX | TRT01A                 |
|----------------|---------------|------|-----|------------------------|
| str            | str           | f64  | str | str                    |
| "CDISCPILOT01" | "01-701-1015" | 63.0 | "F" | "Placebo"              |
| "CDISCPILOT01" | "01-701-1023" | 64.0 | "M" | "Placebo"              |
| "CDISCPILOT01" | "01-701-1028" | 71.0 | "M" | "Xanomeline High Dose" |
| "CDISCPILOT01" | "01-701-1033" | 74.0 | "M" | "Xanomeline Low Dose"  |
| "CDISCPILOT01" | "01-701-1034" | 77.0 | "F" | "Xanomeline High Dose" |


``` python
adsl.filter((pl.col("AGE") >= 75) & (pl.col("SEX") == "F")).select(
    ["USUBJID", "AGE", "SEX", "TRT01A"]
).head()
```


shape: (5, 4)

| USUBJID       | AGE  | SEX | TRT01A                 |
|---------------|------|-----|------------------------|
| str           | f64  | str | str                    |
| "01-701-1034" | 77.0 | "F" | "Xanomeline High Dose" |
| "01-701-1047" | 85.0 | "F" | "Placebo"              |
| "01-701-1111" | 81.0 | "F" | "Xanomeline Low Dose"  |
| "01-701-1133" | 81.0 | "F" | "Xanomeline High Dose" |
| "01-701-1146" | 75.0 | "F" | "Xanomeline High Dose" |


``` python
adsl.filter(pl.col("TRT01A") != "Screen Failure").group_by("TRT01A").agg(
    pl.col("AGE").mean().round(1).alias("mean_age"),
    pl.len().alias("n"),
).sort("TRT01A")
```


shape: (3, 3)

| TRT01A                 | mean_age | n   |
|------------------------|----------|-----|
| str                    | f64      | u32 |
| "Placebo"              | 75.2     | 86  |
| "Xanomeline High Dose" | 73.8     | 72  |
| "Xanomeline Low Dose"  | 76.0     | 96  |


# Read parquet directly

- Dataset label in schema metadata
- Column labels in field metadata.

Both are bytes, decode to str


``` python
from importlib import resources

import pyarrow.parquet as pq

path = resources.files("pharmadata") / "_data" / "pharmaverseadam" / "adsl.parquet"
schema = pq.ParquetFile(path).schema_arrow

print((schema.metadata or {}).get(b"label", b"").decode())
print({
    name: (schema.field(name).metadata or {}).get(b"label", b"").decode()
    for name in schema.names[:3]
})
```


    Subject Level Analysis
    {'STUDYID': 'Study Identifier', 'USUBJID': 'Unique Subject Identifier', 'SUBJID': 'Subject Identifier for the Study'}


``` python
pl.from_arrow(pq.read_table(path, columns=schema.names[:6])).head(5)
```


shape: (5, 6)

| STUDYID        | USUBJID       | SUBJID | SITEID | COUNTRY | RFSTDTC      |
|----------------|---------------|--------|--------|---------|--------------|
| str            | str           | str    | str    | str     | str          |
| "CDISCPILOT01" | "01-701-1015" | "1015" | "701"  | "USA"   | "2014-01-02" |
| "CDISCPILOT01" | "01-701-1023" | "1023" | "701"  | "USA"   | "2012-08-05" |
| "CDISCPILOT01" | "01-701-1028" | "1028" | "701"  | "USA"   | "2013-07-19" |
| "CDISCPILOT01" | "01-701-1033" | "1033" | "701"  | "USA"   | "2014-03-18" |
| "CDISCPILOT01" | "01-701-1034" | "1034" | "701"  | "USA"   | "2014-07-01" |