Last updated: 2023-07-27
Checks: 7 0
Knit directory: muse/
This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.
The command set.seed(20200712) was run prior to running
the code in the R Markdown file. Setting a seed ensures that any results
that rely on randomness, e.g. subsampling or permutations, are
reproducible.
Great job! Recording the operating system, R version, and package versions is critical for reproducibility.
Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.
Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.
The results in this page were generated with repository version 78da4e1. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.
Note that you need to be careful to ensure that all relevant files for
the analysis have been committed to Git prior to generating the results
(you can use wflow_publish or
wflow_git_commit). workflowr only checks the R Markdown
file, but you know if there are other scripts or data files that it
depends on. Below is the status of the Git repository when the results
were generated:
Ignored files:
Ignored: .Rhistory
Ignored: .Rproj.user/
Ignored: r_packages_4.3.0/
Untracked files:
Untracked: analysis/cell_ranger.Rmd
Untracked: analysis/tss_xgboost.Rmd
Untracked: code/multiz100way/
Untracked: data/HG00702_SH089_CHSTrio.chr1.vcf.gz
Untracked: data/HG00702_SH089_CHSTrio.chr1.vcf.gz.tbi
Untracked: data/ncrna_NONCODE[v3.0].fasta.tar.gz
Untracked: data/ncrna_noncode_v3.fa
Untracked: data/netmhciipan.out.gz
Untracked: data/test
Untracked: export/davetang039sblog.WordPress.2023-06-30.xml
Untracked: export/output/
Untracked: women.json
Unstaged changes:
Modified: analysis/graph.Rmd
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the repository in which changes were
made to the R Markdown (analysis/read_files.Rmd) and HTML
(docs/read_files.html) files. If you’ve configured a remote
Git repository (see ?wflow_git_remote), click on the
hyperlinks in the table below to view the files as they were in that
past version.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | 78da4e1 | Dave Tang | 2023-07-27 | Use pbapply |
| html | ccc5fa4 | Dave Tang | 2023-07-26 | Build site. |
| Rmd | b5c3509 | Dave Tang | 2023-07-26 | Read list of files |
I had been using map_dfr from the purrr package to load multiple
files into one single data frame. But this function has been superseded
with the following explanation:
The functions were superseded in purrr 1.0.0 because their names suggest they work like _lgl(), _int(), etc which require length 1 outputs, but actually they return results of any size because the results are combined without any size checks. Additionally, they use dplyr::bind_rows() and dplyr::bind_cols() which require dplyr to be installed and have confusing semantics with edge cases. Superseded functions will not go away, but will only receive critical bug fixes.
I’ll generate some random files to illustrate how
map_dfr is used. I use several packages from the Tidyverse, so if you want to
follow along, you can install them all at once by installing the
tidyverse package.
install.packages("tidyverse")Generate some random files.
random_df <- function(num_row = 100, num_col = 100, seed = 1984){
set.seed(seed)
matrix(
data =
runif(
n = num_row * num_col,
min = 0,
max = 1
),
nrow = num_row
) |> as.data.frame()
}
outdir <- "/tmp/random1984"
random_files <- function(nfiles, prefix = 'x', outdir = 'random', leading_zero = 6){
if(!dir.exists(outdir)){
dir.create(outdir)
}
purrr::map(1:nfiles, function(x){
write.csv(
x = random_df(seed = x),
file = paste0(outdir, '/', prefix, stringr::str_pad(x, leading_zero, pad = 0), ".csv"),
row.names = FALSE
)
}) -> dev_null
}
random_files(50, outdir = outdir)
list.files(outdir) [1] "x000001.csv" "x000002.csv" "x000003.csv" "x000004.csv" "x000005.csv"
[6] "x000006.csv" "x000007.csv" "x000008.csv" "x000009.csv" "x000010.csv"
[11] "x000011.csv" "x000012.csv" "x000013.csv" "x000014.csv" "x000015.csv"
[16] "x000016.csv" "x000017.csv" "x000018.csv" "x000019.csv" "x000020.csv"
[21] "x000021.csv" "x000022.csv" "x000023.csv" "x000024.csv" "x000025.csv"
[26] "x000026.csv" "x000027.csv" "x000028.csv" "x000029.csv" "x000030.csv"
[31] "x000031.csv" "x000032.csv" "x000033.csv" "x000034.csv" "x000035.csv"
[36] "x000036.csv" "x000037.csv" "x000038.csv" "x000039.csv" "x000040.csv"
[41] "x000041.csv" "x000042.csv" "x000043.csv" "x000044.csv" "x000045.csv"
[46] "x000046.csv" "x000047.csv" "x000048.csv" "x000049.csv" "x000050.csv"We can easily load all the files into a single data frame using
map_dfr.
start_time <- Sys.time()
my_df <- map_dfr(list.files(outdir, full.names = TRUE), readr::read_csv, show_col_types = FALSE)
end_time <- Sys.time()
end_time - start_timeTime difference of 31.85373 secsdim(my_df)[1] 5000 100Here’s how to do the same thing using pmap and
bind_rows. (pmap comes with a basic progress
bar, which is nice.) Note that I am using the base R pipe
(|>), which requires R-4.1.0 or higher.
start_time <- Sys.time()
purrr::pmap(
list(list.files(outdir, full.names = TRUE)),
readr::read_csv, show_col_types = FALSE, .progress = FALSE
) |>
dplyr::bind_rows() -> my_df2
end_time <- Sys.time()
end_time - start_timeTime difference of 32.07553 secsall.equal(my_df, my_df2)[1] TRUEOne of the reasons map_dfr was superseded is because it
requires dplyr::bind_rows, which adds a package dependency.
We can use the base R functions do.call and
rbind() instead. In addition, my code above uses
read_csv from the readr package. We can also
substitute that function using the base R read.csv()
function too.
start_time <- Sys.time()
purrr::pmap(
list(list.files(outdir, full.names = TRUE)),
read.csv, .progress = FALSE
) |>
do.call("rbind", args = _) -> my_df3
end_time <- Sys.time()
end_time - start_timeTime difference of 1.201932 secsall.equal(my_df2, my_df3)[1] "Attributes: < Names: 1 string mismatch >"
[2] "Attributes: < Length mismatch: comparison on first 2 components >"
[3] "Attributes: < Component \"class\": Lengths (4, 1) differ (string compare on first 1) >"
[4] "Attributes: < Component \"class\": 1 string mismatch >"
[5] "Attributes: < Component 2: target is externalptr, current is numeric >" The message above from all.equal is saying that the
object attributes are different. We can use the
attributes() function to see the differences.
names(attributes(my_df2))[1] "row.names" "names" "spec" "problems" "class" names(attributes(my_df3))[1] "names" "row.names" "class" Besides the object attributes, the values in the data frames are equal.
We can go one more step in removing the purrr dependency by using
lapply instead. The code below uses all base R functions to
load a list of files.
start_time <- Sys.time()
lapply(
list.files(outdir, full.names = TRUE),
read.csv
) |>
do.call("rbind", args = _) -> my_df4
end_time <- Sys.time()
end_time - start_timeTime difference of 0.7087708 secsall.equal(my_df3, my_df4)[1] TRUEAt this point, you may be wondering whether we needed the Tidyverse packages in the first place. There has already been a lot of discussion on the topic of base R versus Tidyverse, so look it up if you are interested. The point of this post was to illustrate how to read a list of files into a single data frame.
One nice thing about map_dfr is the .id
parameter, which adds an id column that can be useful for distinguishing
the data. The way to use it is to name the input vector.
my_files <- list.files(outdir, full.names = TRUE)
names(my_files) <- sub("\\.\\w+$", "", basename(my_files))
start_time <- Sys.time()
my_df <- map_dfr(my_files, readr::read_csv, show_col_types = FALSE, .id = "file")
end_time <- Sys.time()
end_time - start_timeTime difference of 32.29653 secsmy_df[1:6, 1:6]# A tibble: 6 × 6
file V1 V2 V3 V4 V5
<chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 x000001 0.266 0.655 0.268 0.674 0.659
2 x000001 0.372 0.353 0.219 0.0949 0.185
3 x000001 0.573 0.270 0.517 0.493 0.954
4 x000001 0.908 0.993 0.269 0.462 0.898
5 x000001 0.202 0.633 0.181 0.375 0.944
6 x000001 0.898 0.213 0.519 0.991 0.724This can be achieved using base R as follows.
start_time <- Sys.time()
lapply(
list.files(outdir, full.names = TRUE),
function(x){
cbind(file = sub("\\.\\w+$", "", basename(x)), read.csv(x))
}
) |>
do.call("rbind", args = _) -> my_df5
end_time <- Sys.time()
end_time - start_timeTime difference of 0.9704466 secsmy_df5[1:6, 1:6] file V1 V2 V3 V4 V5
1 x000001 0.2655087 0.6547239 0.2675082 0.67371223 0.6588776
2 x000001 0.3721239 0.3531973 0.2186453 0.09485786 0.1850700
3 x000001 0.5728534 0.2702601 0.5167968 0.49259612 0.9543781
4 x000001 0.9082078 0.9926841 0.2689506 0.46155184 0.8978485
5 x000001 0.2016819 0.6334933 0.1811683 0.37521653 0.9436971
6 x000001 0.8983897 0.2132081 0.5185761 0.99109922 0.7236908We can add a progress bar using the pbapply package. The nice thing about this package is that it supports parallelisation too. (Using parallel here actually slows it down but may be useful when you have a lot of large files.)
library(pbapply)
library(parallel)
cl <- makeCluster(2)
start_time <- Sys.time()
pblapply(
list.files(outdir, full.names = TRUE),
function(x){
cbind(file = sub("\\.\\w+$", "", basename(x)), read.csv(x))
},
cl = cl
) |>
do.call("rbind", args = _) -> my_df6
end_time <- Sys.time()
stopCluster(cl)
end_time - start_timeTime difference of 0.8234155 secsall.equal(my_df5, my_df6)[1] TRUE
sessionInfo()R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: Etc/UTC
tzcode source: system (glibc)
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] pbapply_1.7-0 lubridate_1.9.2 forcats_1.0.0 stringr_1.5.0
[5] dplyr_1.1.2 purrr_1.0.1 readr_2.1.4 tidyr_1.3.0
[9] tibble_3.2.1 ggplot2_3.4.2 tidyverse_2.0.0 workflowr_1.7.0
loaded via a namespace (and not attached):
[1] sass_0.4.6 utf8_1.2.3 generics_0.1.3 stringi_1.7.12
[5] hms_1.1.3 digest_0.6.31 magrittr_2.0.3 timechange_0.2.0
[9] evaluate_0.21 grid_4.3.0 fastmap_1.1.1 rprojroot_2.0.3
[13] jsonlite_1.8.5 processx_3.8.1 whisker_0.4.1 ps_1.7.5
[17] promises_1.2.0.1 httr_1.4.6 fansi_1.0.4 scales_1.2.1
[21] jquerylib_0.1.4 cli_3.6.1 crayon_1.5.2 rlang_1.1.1
[25] bit64_4.0.5 munsell_0.5.0 withr_2.5.0 cachem_1.0.8
[29] yaml_2.3.7 tools_4.3.0 tzdb_0.4.0 colorspace_2.1-0
[33] httpuv_1.6.11 vctrs_0.6.2 R6_2.5.1 lifecycle_1.0.3
[37] git2r_0.32.0 bit_4.0.5 fs_1.6.2 vroom_1.6.3
[41] pkgconfig_2.0.3 callr_3.7.3 pillar_1.9.0 bslib_0.5.0
[45] later_1.3.1 gtable_0.3.3 glue_1.6.2 Rcpp_1.0.10
[49] xfun_0.39 tidyselect_1.2.0 rstudioapi_0.14 knitr_1.43
[53] htmltools_0.5.5 rmarkdown_2.22 compiler_4.3.0 getPass_0.2-2