In order to save time, I want to create a docker image with pre-installed R libraries. -> Big Time saving
One part of the Dockerfile
looks like this
RUN Rscript -e 'require(devtools); devtools::install_version("dplyr", version="0.5.0", repos = "http://cran.rstudio.com/")'
RUN Rscript -e 'library(dplyr); sessionInfo()'
RUN Rscript -e 'require(devtools); devtools::install_version("ggplot2", version="2.2.1", repos = "http://cran.rstudio.com/")'
RUN Rscript -e 'library(dplyr); sessionInfo()'
RUN Rscript -e 'require(devtools); devtools::install_version("knitr", version="1.15.1", repos = "http://cran.rstudio.com/")'
RUN Rscript -e 'library(dplyr); sessionInfo()'
RUN Rscript -e 'require(devtools); devtools::install_version("tidyr", version="0.6.0", repos = "http://cran.rstudio.com/")
Since this is a production environment, I do not want any surprises. Therefore I want to install the same version of my package every single time, hence the specific versions you can see above.
Now my question: If we look at the DESCRIPTION file of the tidyr package (v0.6.0):
https://github.com/tidyverse/tidyr/blob/v0.6.0/DESCRIPTION
It says:
Imports:
tibble,
dplyr (>= 0.4),
stringi,
lazyeval,
magrittr,
Rcpp
In my system, I have dplyr 0.5.0 already preinstalled, so this shouldn’t trigger an update.
This is my docker output (please note I added sessionInfo() statements to display dplyr versions installed) and as you can see dplyr 0.5.0 is installed on the system, and then tidyr version 0.6.0 forces the installation of dplyr, tidyselect and plogr.
Step 56/103 : RUN Rscript -e 'library(dplyr); sessionInfo()'
---> Running in 587c27a05216
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
Warning message:
As of rlang 0.4.0, dplyr must be at least version 0.8.0.
x dplyr 0.5.0 is too old for rlang 0.4.2.
i Please update dplyr with `install.packages("dplyr")`.
R version 3.2.5 (2016-04-14)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
locale:
[1] C
attached base packages:
[1] stats graphics grDevices utils datasets base
other attached packages:
[1] dplyr_0.5.0
loaded via a namespace (and not attached):
[1] magrittr_1.5 R6_2.4.1 assertthat_0.2.1 cli_2.0.0
[5] tools_3.2.5 DBI_1.1.0 pillar_1.4.3 glue_1.3.1
[9] tibble_2.1.3 crayon_1.3.4 Rcpp_1.0.3 fansi_0.4.0
[13] methods_3.2.5 pkgconfig_2.0.3 rlang_0.4.2
---> e55ec5828abb
Removing intermediate container 587c27a05216
Step 57/103 : RUN Rscript -e 'require(devtools); devtools::install_version("tidyr", version="0.6.0", repos = "http://cran.rstudio.com/")'
---> Running in 09ada6215edb
Loading required package: devtools
Loading required package: usethis
Downloading package from url: http://cran.rstudio.com//src/contrib/Archive/tidyr/tidyr_0.6.0.tar.gz
dplyr (0.5.0 -> 0.8.3) [CRAN]
tidyselect (NA -> 0.2.5) [CRAN]
plogr (NA -> 0.2.0) [CRAN]
Installing 3 packages: dplyr, tidyselect, plogr
Installing packages into '/usr/local/r-libs'
(as 'lib' is unspecified)
trying URL 'http://cran.rstudio.com/src/contrib/dplyr_0.8.3.tar.gz'
Content type 'application/x-gzip' length 1173098 bytes (1.1 MB)
==================================================
downloaded 1.1 MB
trying URL 'http://cran.rstudio.com/src/contrib/tidyselect_0.2.5.tar.gz'
Content type 'application/x-gzip' length 21883 bytes (21 KB)
==================================================
downloaded 21 KB
trying URL 'http://cran.rstudio.com/src/contrib/plogr_0.2.0.tar.gz'
Content type 'application/x-gzip' length 7795 bytes
==================================================
downloaded 7795 bytes
* installing *source* package 'tidyselect' ...
** package 'tidyselect' successfully unpacked and MD5 sums checked
** libs
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I/usr/local/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c RcppExports.cpp -o RcppExports.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I/usr/local/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c combine_variables.cpp -o combine_variables.o
g++ -shared -L/usr/local/lib64 -o tidyselect.so RcppExports.o combine_variables.o
installing to /usr/local/r-libs/tidyselect/libs
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (tidyselect)
* installing *source* package 'plogr' ...
** package 'plogr' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (plogr)
* installing *source* package 'dplyr' ...
** package 'dplyr' successfully unpacked and MD5 sums checked
** libs
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c RcppExports.cpp -o RcppExports.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c address.cpp -o address.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c api.cpp -o api.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c arrange.cpp -o arrange.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c between.cpp -o between.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c bind.cpp -o bind.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c distinct.cpp -o distinct.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c encoding.cpp -o encoding.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c filter.cpp -o filter.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c group_indices.cpp -o group_indices.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c hybrid.cpp -o hybrid.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c init.cpp -o init.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c join.cpp -o join.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c join_exports.cpp -o join_exports.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c mutate.cpp -o mutate.o
gcc -std=gnu99 -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -c rlang-export.c -o rlang-export.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c rlang.cpp -o rlang.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c select.cpp -o select.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c set.cpp -o set.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c summarise.cpp -o summarise.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c test.cpp -o test.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c utils-bindings.cpp -o utils-bindings.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c utils.cpp -o utils.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I../inst/include -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_DPLYR -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT -DBOOST_NO_AUTO_PTR -I/usr/local/include -I"/usr/local/r-libs/BH/include" -I"/usr/local/r-libs/plogr/include" -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c window.cpp -o window.o
g++ -shared -L/usr/local/lib64 -o dplyr.so RcppExports.o address.o api.o arrange.o between.o bind.o distinct.o encoding.o filter.o group_indices.o hybrid.o init.o join.o join_exports.o mutate.o rlang-export.o rlang.o select.o set.o summarise.o test.o utils-bindings.o utils.o window.o
installing to /usr/local/r-libs/dplyr/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** preparing package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (dplyr)
The downloaded source packages are in
'/tmp/Rtmp3DsAO3/downloaded_packages'
Installing package into '/usr/local/r-libs'
(as 'lib' is unspecified)
* installing *source* package 'tidyr' ...
** package 'tidyr' successfully unpacked and MD5 sums checked
** libs
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I/usr/local/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c RcppExports.cpp -o RcppExports.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I/usr/local/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c fill.cpp -o fill.o
g++ -I/my-software/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c melt.cpp -o melt.o
g++ -I/my-software/usr/local/bin/lib64/R/include -DNDEBUG -I/usr/local/include -I"/usr/local/r-libs/Rcpp/include" -fpic -O3 -std=gnu++0x -c simplifyPieces.cpp -o simplifyPieces.o
g++ -shared -L/usr/local/lib64 -o tidyr.so RcppExports.o fill.o melt.o simplifyPieces.o
installing to /usr/local/r-libs/tidyr/libs
** R
** data
*** moving datasets to lazyload DB
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (tidyr)
---> 6e55e42fb917
Removing intermediate container 09ada6215edb
Step 58/103 : RUN Rscript -e 'library(dplyr); sessionInfo()'
---> Running in c917b6912a89
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
R version 3.2.5 (2016-04-14)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
locale:
[1] C
attached base packages:
[1] stats graphics grDevices utils datasets base
other attached packages:
[1] dplyr_0.8.3
loaded via a namespace (and not attached):
[1] tidyselect_0.2.5 magrittr_1.5 assertthat_0.2.1 R6_2.4.1
[5] pillar_1.4.3 glue_1.3.1 tibble_2.1.3 crayon_1.3.4
[9] Rcpp_1.0.3 methods_3.2.5 pkgconfig_2.0.3 rlang_0.4.2
[13] purrr_0.3.3
---> b4f3949d3bf8
Removing intermediate container c917b6912a89
The important bit is:
Downloading package from url: http://cran.rstudio.com//src/contrib/Archive/tidyr/tidyr_0.6.0.tar.gz
dplyr (0.5.0 -> 0.8.3) [CRAN]
Where does R find these dependancies as they are not in the DESCRIPTION file (and dplyr v0.4.0 >= is already satisfied)?
2
Answers
Thanks for the previous comment, but I still prefer my old title:
'R package DESCRIPTION file not respected by R ?' (at least the first part).
I managed to resolve the issue, which wasn't related to the warning message of rlang or dplyr, but the line below actually automatically upgrades, which seems default behaviour according to Hadley (see https://github.com/r-lib/devtools/issues/863#issuecomment-137241474)
adjusting this line above to add
(and updating to v1.0.0) actually resolved the issue, as it is now no longer updating packages which are already satisfied. (see new line below)
I understand this isn't really an 'answer' since it's expected behaviour according to the commenter in the link, but still wanted to post it here for people running into the same problem in the future.
My main issue is that from a philosophy perspective, I would understand that install.packages() would install the latest version of an R package. But if devtools::install_version() is used, it's likely because you want to install a specific version of software. In that scenario, does it not make sense to NOT upgrade the dependancies ? This links back to the original title, which is that if the dependancies are satisfied according to the DEPENDANCIES file, should R really force an upgrade by default?
Part of your output tells you the answer
If you want to use rlang 0.4.0 you need to have at least dplyr 0.8.0.
Sidenote
Please use fewer docker layers, especially if this image will be utilized in production. Or squash the layers at the end.
There is a flag to squash when building your image. It is documented here
The gist of it
For compatibility across various infrastructure setups, I would recommend not to exceed 42 image layers. Here the origin of the statement regarding 42 layers:
https://stackoverflow.com/a/39383801/12153397