{ "cells": [ { "cell_type": "markdown", "id": "137a6624-8324-4ae1-b8a0-e783b4b7bf47", "metadata": {}, "source": [ "# Step 2: Normalize your data using sctransform\n", "\n", "Use this notebook to normalize your data using the [sctransform](https://satijalab.org/seurat/articles/sctransform_vignette) package. \n", "\n", "Please note that the sctransform package is written in R. Therefore, in order to run this notebook you will first need to [install R]() on your system, as well as [sctransform version 0.4.1](https://github.com/satijalab/sctransform) and it's corresponding dependencies (including the optional dependency [glmGamPoi](https://github.com/const-ae/glmGamPoi)). \n", "\n", "Also note that your raw data input should be a csv file arranged using [tidy format](https://tidyr.tidyverse.org/articles/tidy-data.html). At a minimum your input csv should have five columns:\n", "1. A column that corresponds to the first mode of your tensor. In metatranscriptomic data this column might indicate gene ID.\n", " - This first mode should generally be the longest in your tensor, and the one that corresponds to the variable you want clustered (e.g. genes in the case of metatranscriptomics data). The sparsity penalty (`lambda`) will be applied to this mode.\n", "1. A column that corresponds to the second mode of your tensor. In metatranscriptomic data this column might indicate taxon ID.\n", "1. A column that corresponds to the third mode of your tensor. This column should indicate sample ID.\n", " - **IMPORTANT: Sample IDs should be identical for different replicates of the same sample condition (see example below).**\n", "1. A column that indicates the replicate ID of the sample.\n", "1. A column that corresponds to the data variable. For raw metatranscriptomic data this column might contain read counts.\n", "\n", "Here's a snippet of how an example csv might be arranged:\n", "\n", "| gene_id | taxon_id | sample_id | replicate | read_counts |\n", "|---------|------------|-----------|-----------|-------------|\n", "| K03839 | P. marinus | sample1 | A | 3.02 |\n", "| K03839 | P. marinus | sample1 | B | 3.31 |\n", "| K03839 | P. marinus | sample1 | C | 3.18 |\n", "| K03839 | P. marinus | sample2 | A | 10.40 |\n", "| ... | ... | ... | ... | ... |\n", "| K03320 | S. marinus | sample9 | C | 0.05 |" ] }, { "cell_type": "code", "execution_count": 29, "id": "086abf9f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The rpy2.ipython extension is already loaded. To reload it, use:\n", " %reload_ext rpy2.ipython\n" ] } ], "source": [ "# imports\n", "\n", "# python packages\n", "import math\n", "import numpy as np\n", "import os\n", "import pandas as pd\n", "import rpy2\n", "import seaborn as sns\n", "\n", "from matplotlib import pyplot as plt\n", "\n", "# rpy2 imports\n", "from rpy2 import robjects as ro\n", "from rpy2.robjects.packages import importr\n", "from rpy2.ipython.ggplot import image_png\n", "from rpy2.robjects import pandas2ri\n", "from rpy2.robjects.conversion import localconverter\n", "\n", "# load rpy2 extension for ipython\n", "%load_ext rpy2.ipython\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "acbf764a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting rpy2\n", " Downloading rpy2-3.6.4-py3-none-any.whl.metadata (5.4 kB)\n", "Collecting rpy2-rinterface>=3.6.3 (from rpy2)\n", " Downloading rpy2_rinterface-3.6.3.tar.gz (79 kB)\n", " Installing build dependencies ... \u001b[?25ldone\n", "\u001b[?25h Getting requirements to build wheel ... \u001b[?25ldone\n", "\u001b[?25h Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n", "\u001b[?25hCollecting rpy2-robjects>=3.6.3 (from rpy2)\n", " Downloading rpy2_robjects-3.6.3-py3-none-any.whl.metadata (3.3 kB)\n", "Requirement already satisfied: cffi>=1.15.1 in /home/sr320/miniconda3/envs/barnacle311/lib/python3.11/site-packages (from rpy2-rinterface>=3.6.3->rpy2) (1.16.0)\n", "Requirement already satisfied: pycparser in /home/sr320/miniconda3/envs/barnacle311/lib/python3.11/site-packages (from cffi>=1.15.1->rpy2-rinterface>=3.6.3->rpy2) (2.22)\n", "Requirement already satisfied: jinja2 in /home/sr320/miniconda3/envs/barnacle311/lib/python3.11/site-packages (from rpy2-robjects>=3.6.3->rpy2) (3.1.4)\n", "Collecting tzlocal (from rpy2-robjects>=3.6.3->rpy2)\n", " Downloading tzlocal-5.3.1-py3-none-any.whl.metadata (7.6 kB)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /home/sr320/miniconda3/envs/barnacle311/lib/python3.11/site-packages (from jinja2->rpy2-robjects>=3.6.3->rpy2) (2.1.5)\n", "Downloading rpy2-3.6.4-py3-none-any.whl (9.9 kB)\n", "Downloading rpy2_robjects-3.6.3-py3-none-any.whl (125 kB)\n", "Downloading tzlocal-5.3.1-py3-none-any.whl (18 kB)\n", "Building wheels for collected packages: rpy2-rinterface\n", " Building wheel for rpy2-rinterface (pyproject.toml) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for rpy2-rinterface: filename=rpy2_rinterface-3.6.3-cp311-cp311-linux_x86_64.whl size=140348 sha256=71ecbf2f47de042f7e061a8fb06683cf6a5837fd7c1e69ae8f15abc29f5f22a0\n", " Stored in directory: /home/sr320/.cache/pip/wheels/9c/70/19/948292388eaaf757c279a12add5d0c5a886078cd51aa39a69f\n", "Successfully built rpy2-rinterface\n", "Installing collected packages: tzlocal, rpy2-rinterface, rpy2-robjects, rpy2\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4/4\u001b[0m [rpy2][32m2/4\u001b[0m [rpy2-robjects]e]\n", "\u001b[1A\u001b[2KSuccessfully installed rpy2-3.6.4 rpy2-rinterface-3.6.3 rpy2-robjects-3.6.3 tzlocal-5.3.1\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "# Install required packages\n", "%pip install rpy2" ] }, { "cell_type": "code", "execution_count": 30, "id": "7c54814d", "metadata": {}, "outputs": [ { "ename": "RNotReadyError", "evalue": "Embedded R is not ready to use.", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/sexp.py:634\u001b[0m, in \u001b[0;36mSexpVectorAbstract.from_object\u001b[0;34m(cls, obj)\u001b[0m\n\u001b[1;32m 633\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 634\u001b[0m mv \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mmemoryview\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 635\u001b[0m res \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39mfrom_memoryview(mv)\n", "\u001b[0;31mTypeError\u001b[0m: memoryview: a bytes-like object is required, not 'tuple'", "\nDuring handling of the above exception, another exception occurred:\n", "\u001b[0;31mRNotReadyError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[30], line 4\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# install & import sctransform and other r package dependencies\u001b[39;00m\n\u001b[1;32m 2\u001b[0m \n\u001b[1;32m 3\u001b[0m \u001b[38;5;66;03m# check if sctransform is installed\u001b[39;00m\n\u001b[0;32m----> 4\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[43mro\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpackages\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43misinstalled\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msctransform\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m:\n\u001b[1;32m 5\u001b[0m \u001b[38;5;66;03m# select CRAN mirror\u001b[39;00m\n\u001b[1;32m 6\u001b[0m utils \u001b[38;5;241m=\u001b[39m importr(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mutils\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 7\u001b[0m utils\u001b[38;5;241m.\u001b[39mchooseCRANmirror(ind\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/robjects/packages.py:428\u001b[0m, in \u001b[0;36misinstalled\u001b[0;34m(name, lib_loc)\u001b[0m\n\u001b[1;32m 419\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 420\u001b[0m \u001b[38;5;124;03mFind whether an R package is installed\u001b[39;00m\n\u001b[1;32m 421\u001b[0m \u001b[38;5;124;03m:param name: name of an R package\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 424\u001b[0m \u001b[38;5;124;03m:rtype: a :class:`bool`\u001b[39;00m\n\u001b[1;32m 425\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 427\u001b[0m instapack \u001b[38;5;241m=\u001b[39m InstalledPackages(lib_loc)\n\u001b[0;32m--> 428\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43minstapack\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43misinstalled\u001b[49m\u001b[43m(\u001b[49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/robjects/packages.py:395\u001b[0m, in \u001b[0;36mInstalledPackages.isinstalled\u001b[0;34m(self, packagename)\u001b[0m\n\u001b[1;32m 393\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21misinstalled\u001b[39m(\u001b[38;5;28mself\u001b[39m, packagename: \u001b[38;5;28mstr\u001b[39m):\n\u001b[1;32m 394\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(packagename, rinterface\u001b[38;5;241m.\u001b[39mStrSexpVector):\n\u001b[0;32m--> 395\u001b[0m \u001b[43mrinterface\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mStrSexpVector\u001b[49m\u001b[43m(\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpackagename\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 396\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 397\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(packagename) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/sexp.py:769\u001b[0m, in \u001b[0;36mSexpVector.__init__\u001b[0;34m(self, obj)\u001b[0m\n\u001b[1;32m 767\u001b[0m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(obj)\n\u001b[1;32m 768\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(obj, collections\u001b[38;5;241m.\u001b[39mabc\u001b[38;5;241m.\u001b[39mSized):\n\u001b[0;32m--> 769\u001b[0m robj: Sexp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mtype\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_object\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 770\u001b[0m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39m\u001b[38;5;21m__init__\u001b[39m(robj)\n\u001b[1;32m 771\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/sexp.py:638\u001b[0m, in \u001b[0;36mSexpVectorAbstract.from_object\u001b[0;34m(cls, obj)\u001b[0m\n\u001b[1;32m 636\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mTypeError\u001b[39;00m, \u001b[38;5;167;01mValueError\u001b[39;00m):\n\u001b[1;32m 637\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 638\u001b[0m res \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_iterable\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 639\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m:\n\u001b[1;32m 640\u001b[0m msg \u001b[38;5;241m=\u001b[39m (\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mThe class methods from_memoryview() and \u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m 641\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfrom_iterable() both failed to make a \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m 642\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfrom an object of class \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m 643\u001b[0m \u001b[38;5;241m.\u001b[39mformat(\u001b[38;5;28mcls\u001b[39m, \u001b[38;5;28mtype\u001b[39m(obj)))\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/conversion.py:46\u001b[0m, in \u001b[0;36m_cdata_res_to_rinterface.._\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 45\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m---> 46\u001b[0m cdata \u001b[38;5;241m=\u001b[39m \u001b[43mfunction\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 47\u001b[0m \u001b[38;5;66;03m# TODO: test cdata is of the expected CType\u001b[39;00m\n\u001b[1;32m 48\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m _cdata_to_rinterface(cdata)\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/sexp.py:559\u001b[0m, in \u001b[0;36mSexpVectorAbstract.from_iterable\u001b[0;34m(cls, iterable, populate_func, set_elt, cast_value)\u001b[0m\n\u001b[1;32m 557\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Create an R vector/array from an iterable.\"\"\"\u001b[39;00m\n\u001b[1;32m 558\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m embedded\u001b[38;5;241m.\u001b[39misready():\n\u001b[0;32m--> 559\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m embedded\u001b[38;5;241m.\u001b[39mRNotReadyError(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mEmbedded R is not ready to use.\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 560\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m populate_func \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 561\u001b[0m populate_func \u001b[38;5;241m=\u001b[39m _populate_r_vector\n", "\u001b[0;31mRNotReadyError\u001b[0m: Embedded R is not ready to use." ] } ], "source": [ "# install & import sctransform and other r package dependencies\n", "\n", "# check if sctransform is installed\n", "if not ro.packages.isinstalled('sctransform'):\n", " # select CRAN mirror\n", " utils = importr('utils')\n", " utils.chooseCRANmirror(ind=1)\n", " # install sctransform\n", " utils.install_packages(ro.vectors.StrVector(['sctransform']))\n", "\n", "# import sctransform and R Matrix package\n", "sctransform = importr('sctransform')\n", "rmatrix = importr('Matrix')\n", "\n", "# check sctransform version (should be 0.4.1 or compatible)\n", "print(f'Installed sctransform version: {sctransform.__version__}')\n", "if not sctransform.__version__ in ['0.4.1', '0.4.2']:\n", " raise Exception('Please ensure that the installed sctransform is version 0.4.1 or 0.4.2')\n", " \n", "# check that glmGamPoi dependency is installed\n", "if not ro.packages.isinstalled('glmGamPoi'):\n", " print('Installing glmGamPoi dependency...')\n", " # install glmGamPoi from Bioconductor non-interactively\n", " utils = importr('utils')\n", " utils.install_packages(ro.vectors.StrVector(['BiocManager']))\n", " biocmanager = importr('BiocManager')\n", " biocmanager.install(ro.vectors.StrVector(['glmGamPoi']), ask=False, update=False)\n", " print('glmGamPoi installed successfully')\n" ] }, { "cell_type": "code", "execution_count": 23, "id": "d807f0b5", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "R callback write-console: Installing package into ‘/home/sr320/R/x86_64-pc-linux-gnu-library/4.2’\n", "(as ‘lib’ is unspecified)\n", " \n", "R callback write-console: trying URL 'https://cran.r-project.org/src/contrib/Archive/Matrix/Matrix_1.5-3.tar.gz'\n", " \n", "R callback write-console: Content type 'application/x-gzip' \n", "R callback write-console: length 2163568 bytes (2.1 MB)\n", " \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: \n", " \n", "R callback write-console: downloaded 2.1 MB\n", "\n", " \n", "* installing *source* package ‘Matrix’ ...\n", "** package ‘Matrix’ successfully unpacked and MD5 sums checked\n", "** using staged installation\n", "** libs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c CHMfactor.c -o CHMfactor.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c Csparse.c -o Csparse.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c init.c -o init.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c Mutils.c -o Mutils.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c chm_common.c -o chm_common.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c cs.c -o cs.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c cs_utils.c -o cs_utils.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dense.c -o dense.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dgCMatrix.c -o dgCMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dgeMatrix.c -o dgeMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dpoMatrix.c -o dpoMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dppMatrix.c -o dppMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dsCMatrix.c -o dsCMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dsyMatrix.c -o dsyMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dspMatrix.c -o dspMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dtCMatrix.c -o dtCMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dtrMatrix.c -o dtrMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c dtpMatrix.c -o dtpMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c factorizations.c -o factorizations.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c sparseQR.c -o sparseQR.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c sparseVector.c -o sparseVector.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c abIndex.c -o abIndex.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c packedMatrix.c -o packedMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c unpackedMatrix.c -o unpackedMatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c sparse.c -o sparse.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I./SuiteSparse_config -DUSE_FC_LEN_T -I/usr/local/include -fpic -g -O2 -c validity.c -o validity.o\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD'\n", "( cd Lib ; make clean )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD/Lib'\n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD/Lib'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD'\n", "( cd Source ; make clean )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD/Source'\n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD/Source'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD'\n", "( cd Source ; make clean )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD/Source'\n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD/Source'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/SuiteSparse_config'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/SuiteSparse_config'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD'\n", "( cd Lib ; make -f \"/opt/R/4.2.3/lib/R/etc/Makeconf\" -f Makefile )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD/Lib'\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_aat.c -o cholmod_aat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_add.c -o cholmod_add.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_band.c -o cholmod_band.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_change_factor.c -o cholmod_change_factor.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_common.c -o cholmod_common.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_complex.c -o cholmod_complex.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_copy.c -o cholmod_copy.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_dense.c -o cholmod_dense.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_error.c -o cholmod_error.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_factor.c -o cholmod_factor.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_memory.c -o cholmod_memory.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_sparse.c -o cholmod_sparse.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_transpose.c -o cholmod_transpose.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_triplet.c -o cholmod_triplet.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Core/cholmod_version.c -o cholmod_version.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Check/cholmod_check.c -o cholmod_check.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Check/cholmod_read.c -o cholmod_read.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Check/cholmod_write.c -o cholmod_write.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_amd.c -o cholmod_amd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_analyze.c -o cholmod_analyze.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_colamd.c -o cholmod_colamd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_etree.c -o cholmod_etree.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_factorize.c -o cholmod_factorize.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_postorder.c -o cholmod_postorder.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_rcond.c -o cholmod_rcond.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_resymbol.c -o cholmod_resymbol.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_rowcolcounts.c -o cholmod_rowcolcounts.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_rowfac.c -o cholmod_rowfac.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_solve.c -o cholmod_solve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Cholesky/cholmod_spsolve.c -o cholmod_spsolve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_drop.c -o cholmod_drop.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_horzcat.c -o cholmod_horzcat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_norm.c -o cholmod_norm.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_scale.c -o cholmod_scale.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_sdmult.c -o cholmod_sdmult.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_ssmult.c -o cholmod_ssmult.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_submatrix.c -o cholmod_submatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_vertcat.c -o cholmod_vertcat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../MatrixOps/cholmod_symmetry.c -o cholmod_symmetry.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Modify/cholmod_rowadd.c -o cholmod_rowadd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Modify/cholmod_rowdel.c -o cholmod_rowdel.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Modify/cholmod_updown.c -o cholmod_updown.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Supernodal/cholmod_super_numeric.c -o cholmod_super_numeric.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Supernodal/cholmod_super_solve.c -o cholmod_super_solve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -c ../Supernodal/cholmod_super_symbolic.c -o cholmod_super_symbolic.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_aat.c -o cholmod_l_aat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_add.c -o cholmod_l_add.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_band.c -o cholmod_l_band.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_change_factor.c -o cholmod_l_change_factor.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_common.c -o cholmod_l_common.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_complex.c -o cholmod_l_complex.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_copy.c -o cholmod_l_copy.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_dense.c -o cholmod_l_dense.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_error.c -o cholmod_l_error.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_factor.c -o cholmod_l_factor.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_memory.c -o cholmod_l_memory.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_sparse.c -o cholmod_l_sparse.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_transpose.c -o cholmod_l_transpose.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_triplet.c -o cholmod_l_triplet.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Core/cholmod_version.c -o cholmod_l_version.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Check/cholmod_check.c -o cholmod_l_check.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Check/cholmod_read.c -o cholmod_l_read.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Check/cholmod_write.c -o cholmod_l_write.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_amd.c -o cholmod_l_amd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_analyze.c -o cholmod_l_analyze.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_colamd.c -o cholmod_l_colamd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_etree.c -o cholmod_l_etree.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_factorize.c -o cholmod_l_factorize.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_postorder.c -o cholmod_l_postorder.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_rcond.c -o cholmod_l_rcond.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_resymbol.c -o cholmod_l_resymbol.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_rowcolcounts.c -o cholmod_l_rowcolcounts.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_rowfac.c -o cholmod_l_rowfac.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_solve.c -o cholmod_l_solve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Cholesky/cholmod_spsolve.c -o cholmod_l_spsolve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_drop.c -o cholmod_l_drop.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_horzcat.c -o cholmod_l_horzcat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_norm.c -o cholmod_l_norm.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_scale.c -o cholmod_l_scale.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_sdmult.c -o cholmod_l_sdmult.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_ssmult.c -o cholmod_l_ssmult.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_submatrix.c -o cholmod_l_submatrix.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_vertcat.c -o cholmod_l_vertcat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../MatrixOps/cholmod_symmetry.c -o cholmod_l_symmetry.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Modify/cholmod_rowadd.c -o cholmod_l_rowadd.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Modify/cholmod_rowdel.c -o cholmod_l_rowdel.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Modify/cholmod_updown.c -o cholmod_l_updown.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Supernodal/cholmod_super_numeric.c -o cholmod_l_super_numeric.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Supernodal/cholmod_super_solve.c -o cholmod_l_super_solve.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../../AMD/Include -I../../AMD/Source -I../../COLAMD/Include -I../Include -I../../SuiteSparse_config -DNPARTITION -DNPRINT -I/usr/local/include -fpic -g -O2 -DDLONG -c ../Supernodal/cholmod_super_symbolic.c -o cholmod_l_super_symbolic.o\n", "ar -rucs ../../CHOLMOD.a cholmod_aat.o cholmod_add.o cholmod_band.o cholmod_change_factor.o cholmod_common.o cholmod_complex.o cholmod_copy.o cholmod_dense.o cholmod_error.o cholmod_factor.o cholmod_memory.o cholmod_sparse.o cholmod_transpose.o cholmod_triplet.o cholmod_version.o cholmod_check.o cholmod_read.o cholmod_write.o cholmod_amd.o cholmod_analyze.o cholmod_colamd.o cholmod_etree.o cholmod_factorize.o cholmod_postorder.o cholmod_rcond.o cholmod_resymbol.o cholmod_rowcolcounts.o cholmod_rowfac.o cholmod_solve.o cholmod_spsolve.o cholmod_drop.o cholmod_horzcat.o cholmod_norm.o cholmod_scale.o cholmod_sdmult.o cholmod_ssmult.o cholmod_submatrix.o cholmod_vertcat.o cholmod_symmetry.o cholmod_rowadd.o cholmod_rowdel.o cholmod_updown.o cholmod_super_numeric.o cholmod_super_solve.o cholmod_super_symbolic.o cholmod_l_aat.o cholmod_l_add.o cholmod_l_band.o cholmod_l_change_factor.o cholmod_l_common.o cholmod_l_complex.o cholmod_l_copy.o cholmod_l_dense.o cholmod_l_error.o cholmod_l_factor.o cholmod_l_memory.o cholmod_l_sparse.o cholmod_l_transpose.o cholmod_l_triplet.o cholmod_l_version.o cholmod_l_check.o cholmod_l_read.o cholmod_l_write.o cholmod_l_amd.o cholmod_l_analyze.o cholmod_l_colamd.o cholmod_l_etree.o cholmod_l_factorize.o cholmod_l_postorder.o cholmod_l_rcond.o cholmod_l_resymbol.o cholmod_l_rowcolcounts.o cholmod_l_rowfac.o cholmod_l_solve.o cholmod_l_spsolve.o cholmod_l_drop.o cholmod_l_horzcat.o cholmod_l_norm.o cholmod_l_scale.o cholmod_l_sdmult.o cholmod_l_ssmult.o cholmod_l_submatrix.o cholmod_l_vertcat.o cholmod_l_symmetry.o cholmod_l_rowadd.o cholmod_l_rowdel.o cholmod_l_updown.o cholmod_l_super_numeric.o cholmod_l_super_solve.o cholmod_l_super_symbolic.o \n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD/Lib'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/CHOLMOD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD'\n", "( cd Source ; make -f \"/opt/R/4.2.3/lib/R/etc/Makeconf\" -f Makefile lib )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD/Source'\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -c colamd.c -o colamd.o\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "ar: `u' modifier ignored since `D' is the default (see `U')\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c colamd.c -o colamd_l.o\n", "ar -rucs ../../COLAMD.a colamd.o colamd_l.o\n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD/Source'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/COLAMD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD'\n", "( cd Source ; make -f \"/opt/R/4.2.3/lib/R/etc/Makeconf\" -f Makefile lib )\n", "make[2]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD/Source'\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_aat.c -o amd_i_aat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_aat.c -o amd_l_aat.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_1.c -o amd_i_1.o\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "ar: `u' modifier ignored since `D' is the default (see `U')\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_1.c -o amd_l_1.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_2.c -o amd_i_2.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_2.c -o amd_l_2.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_postorder.c -o amd_i_postorder.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_postorder.c -o amd_l_postorder.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_post_tree.c -o amd_i_post_tree.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_post_tree.c -o amd_l_post_tree.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_defaults.c -o amd_i_defaults.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_defaults.c -o amd_l_defaults.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_order.c -o amd_i_order.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_order.c -o amd_l_order.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_control.c -o amd_i_control.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_control.c -o amd_l_control.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_info.c -o amd_i_info.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_info.c -o amd_l_info.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_valid.c -o amd_i_valid.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_valid.c -o amd_l_valid.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_preprocess.c -o amd_i_preprocess.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_preprocess.c -o amd_l_preprocess.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDINT -c amd_dump.c -o amd_i_dump.o\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I../Include -I../../SuiteSparse_config -I/usr/local/include -fpic -g -O2 -I../Include -DDLONG -c amd_dump.c -o amd_l_dump.o\n", "ar -rucs ../../AMD.a amd_i_aat.o amd_l_aat.o amd_i_1.o amd_l_1.o amd_i_2.o amd_l_2.o amd_i_postorder.o amd_l_postorder.o amd_i_post_tree.o amd_l_post_tree.o amd_i_defaults.o amd_l_defaults.o amd_i_order.o amd_l_order.o amd_i_control.o amd_l_control.o amd_i_info.o amd_l_info.o amd_i_valid.o amd_l_valid.o amd_i_preprocess.o amd_l_preprocess.o amd_i_dump.o amd_l_dump.o\n", "make[2]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD/Source'\n", "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/AMD'\n", "make[1]: Entering directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/SuiteSparse_config'\n", "gcc -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -DNTIMER -I/usr/local/include -fpic -g -O2 -c SuiteSparse_config.c -o SuiteSparse_config.o\n", "ar -rucs ../SuiteSparse_config.a SuiteSparse_config.o\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "ar: `u' modifier ignored since `D' is the default (see `U')\n", "ar: `u' modifier ignored since `D' is the default (see `U')\n", "installing to /home/sr320/R/x86_64-pc-linux-gnu-library/4.2/00LOCK-Matrix/00new/Matrix/libs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "make[1]: Leaving directory '/tmp/RtmpYYRdq5/R.INSTALL3980a7413436a/Matrix/src/SuiteSparse_config'\n", "gcc -shared -L/opt/R/4.2.3/lib/R/lib -L/usr/local/lib -o Matrix.so CHMfactor.o Csparse.o init.o Mutils.o chm_common.o cs.o cs_utils.o dense.o dgCMatrix.o dgeMatrix.o dpoMatrix.o dppMatrix.o dsCMatrix.o dsyMatrix.o dspMatrix.o dtCMatrix.o dtrMatrix.o dtpMatrix.o factorizations.o sparseQR.o sparseVector.o abIndex.o packedMatrix.o unpackedMatrix.o sparse.o validity.o CHOLMOD.a COLAMD.a AMD.a SuiteSparse_config.a -llapack -lblas -lgfortran -lm -lquadmath -L/opt/R/4.2.3/lib/R/lib -lR\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "** R\n", "** data\n", "** inst\n", "** byte-compile and prepare package for lazy loading\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Creating a generic function for ‘zapsmall’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘drop’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘cov2cor’ from package ‘stats’ in package ‘Matrix’\n", "Creating a generic function for ‘unname’ from package ‘base’ in package ‘Matrix’\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"dgCMatrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"dgRMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"dgTMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"dgCMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"dgCMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"CsparseMatrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"RsparseMatrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"TsparseMatrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"CsparseMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"RsparseMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"TsparseMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"CsparseMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"RsparseMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"TsparseMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"sparseMatrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"sparseMatrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"sparseMatrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"matrix.csc\",\"Matrix\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"matrix.csr\",\"Matrix\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"matrix.coo\",\"Matrix\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"dgCMatrix\",\"matrix.csc\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"dgRMatrix\",\"matrix.csr\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"dgTMatrix\",\"matrix.coo\"’: no definition for class “matrix.coo”\n", "in method for ‘coerce’ with signature ‘\"Matrix\",\"matrix.csc\"’: no definition for class “matrix.csc”\n", "in method for ‘coerce’ with signature ‘\"Matrix\",\"matrix.csr\"’: no definition for class “matrix.csr”\n", "in method for ‘coerce’ with signature ‘\"Matrix\",\"matrix.coo\"’: no definition for class “matrix.coo”\n", "Creating a generic function for ‘chol2inv’ from ‘base’ in package ‘Matrix’\n", " (from the saved implicit definition)\n", "Creating a generic function for ‘chol2inv’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘colSums’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘colMeans’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘rowSums’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘rowMeans’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘diag’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘diag<-’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘crossprod’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘tcrossprod’ from package ‘base’ in package ‘Matrix’\n", "in method for ‘coerce’ with signature ‘\"graphAM\",\"TsparseMatrix\"’: no definition for class “graphAM”\n", "in method for ‘coerce’ with signature ‘\"graphNEL\",\"TsparseMatrix\"’: no definition for class “graphNEL”\n", "in method for ‘coerce’ with signature ‘\"graph\",\"CsparseMatrix\"’: no definition for class “graph”\n", "in method for ‘coerce’ with signature ‘\"graph\",\"RsparseMatrix\"’: no definition for class “graph”\n", "in method for ‘coerce’ with signature ‘\"graph\",\"TsparseMatrix\"’: no definition for class “graph”\n", "in method for ‘coerce’ with signature ‘\"graph\",\"sparseMatrix\"’: no definition for class “graph”\n", "in method for ‘coerce’ with signature ‘\"graph\",\"Matrix\"’: no definition for class “graph”\n", "in method for ‘coerce’ with signature ‘\"TsparseMatrix\",\"graphNEL\"’: no definition for class “graphNEL”\n", "in method for ‘coerce’ with signature ‘\"Matrix\",\"graphNEL\"’: no definition for class “graphNEL”\n", "in method for ‘coerce’ with signature ‘\"Matrix\",\"graph\"’: no definition for class “graph”\n", "Creating a generic function for ‘norm’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘rcond’ from ‘base’ in package ‘Matrix’\n", " (from the saved implicit definition)\n", "Creating a generic function for ‘rcond’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.R’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.Q’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.qy’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.qty’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.coef’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.resid’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘qr.fitted’ from package ‘base’ in package ‘Matrix’\n", "Creating a generic function for ‘toeplitz’ from package ‘stats’ in package ‘Matrix’\n", "Creating a generic function for ‘which’ from package ‘base’ in package ‘Matrix’\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "** help\n", "*** installing help indices\n", "** building package indices\n", "** installing vignettes\n", "** testing if installed package can be loaded from temporary location\n", "** checking absolute paths in shared objects and dynamic libraries\n", "** testing if installed package can be loaded from final location\n", "** testing if installed package keeps a record of temporary installation path\n", "* DONE (Matrix)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Matrix package downgraded successfully\n" ] } ], "source": [ "# Fix Matrix package compatibility issue\n", "try:\n", " # Try to fix the Matrix package compatibility issue\n", " utils = importr('utils')\n", " utils.install_packages(\"https://cran.r-project.org/src/contrib/Archive/Matrix/Matrix_1.5-3.tar.gz\", repos=ro.NULL, type=\"source\")\n", " rmatrix = importr('Matrix')\n", " print(\"Matrix package downgraded successfully\")\n", "except Exception as e:\n", " print(f\"Could not downgrade Matrix package: {e}\")\n", " print(\"Trying to continue with current version...\")" ] }, { "cell_type": "code", "execution_count": 26, "id": "ae0a2f0a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "R callback write-console: Installing package into ‘/home/sr320/R/x86_64-pc-linux-gnu-library/4.2’\n", "(as ‘lib’ is unspecified)\n", " \n", "R callback write-console: trying URL 'https://cran.r-project.org/src/contrib/Archive/sctransform/sctransform_0.4.1.tar.gz'\n", " \n", "R callback write-console: Content type 'application/x-gzip' \n", "R callback write-console: length 197540 bytes (192 KB)\n", " \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: = \n", "R callback write-console: \n", " \n", "R callback write-console: downloaded 192 KB\n", "\n", " \n", "* installing *source* package ‘sctransform’ ...\n", "** package ‘sctransform’ successfully unpacked and MD5 sums checked\n", "** using staged installation\n", "** libs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "g++ -std=gnu++17 -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I'/home/sr320/R/x86_64-pc-linux-gnu-library/4.2/RcppArmadillo/include' -I'/home/sr320/R/x86_64-pc-linux-gnu-library/4.2/Rcpp/include' -I/usr/local/include -fpic -g -O2 -c RcppExports.cpp -o RcppExports.o\n", "g++ -std=gnu++17 -I\"/opt/R/4.2.3/lib/R/include\" -DNDEBUG -I'/home/sr320/R/x86_64-pc-linux-gnu-library/4.2/RcppArmadillo/include' -I'/home/sr320/R/x86_64-pc-linux-gnu-library/4.2/Rcpp/include' -I/usr/local/include -fpic -g -O2 -c utils.cpp -o utils.o\n", "g++ -std=gnu++17 -shared -L/opt/R/4.2.3/lib/R/lib -L/usr/local/lib -o sctransform.so RcppExports.o utils.o -llapack -lblas -lgfortran -lm -lquadmath -L/opt/R/4.2.3/lib/R/lib -lR\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "installing to /home/sr320/R/x86_64-pc-linux-gnu-library/4.2/00LOCK-sctransform/00new/sctransform/libs\n", "** R\n", "** data\n", "*** moving datasets to lazyload DB\n", "** inst\n", "** byte-compile and prepare package for lazy loading\n", "** help\n", "*** installing help indices\n", "** building package indices\n", "** testing if installed package can be loaded from temporary location\n", "** checking absolute paths in shared objects and dynamic libraries\n", "** testing if installed package can be loaded from final location\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "sctransform 0.4.1 installed successfully\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "** testing if installed package keeps a record of temporary installation path\n", "* DONE (sctransform)\n" ] } ], "source": [ "# Install sctransform 0.4.1 specifically\n", "try:\n", " utils = importr('utils')\n", " # Install specific version of sctransform\n", " utils.install_packages(\"https://cran.r-project.org/src/contrib/Archive/sctransform/sctransform_0.4.1.tar.gz\", \n", " repos=ro.NULL, type=\"source\")\n", " print(\"sctransform 0.4.1 installed successfully\")\n", "except Exception as e:\n", " print(f\"Could not install sctransform 0.4.1: {e}\")\n", " print(\"Continuing with current version...\")" ] }, { "cell_type": "code", "execution_count": 31, "id": "0182ee2a", "metadata": {}, "outputs": [], "source": [ "# Restart R session and reload\n", "import rpy2.rinterface as rinterface\n", "# Restart the embedded R instance\n", "%reload_ext rpy2.ipython" ] }, { "cell_type": "code", "execution_count": null, "id": "3179d24a", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "26c3bee8", "metadata": {}, "source": [ "### Input data\n", "\n", "In this step you will enter the variables necessary to:\n", "1. Locate your input data\n", "1. Configure your data tensor (i.e. which three variables correspond to the three different modes or axes)\n", "1. Configure your normalization scheme\n", "\n", "When it comes to normalization, there are several parameters you need to define:\n", "- You will normalize by one of the variables that corresponds to a mode in your data tensor. You can think of this as the variable that defines the \"within\" groups. For example if I am normalizing by taxa, then the normalization procedure will be applied independently within each taxon in the dataset. Put another way, you can think of each taxon as a slab of your data tensor (i.e. a matrix), and the normalization will be applied independently to each slab.\n", "- For each slab, you will define which of the remaining two modes best corresponds to samples, and which to genes. For example, in metatranscriptomic data annotated with KEGG orthologies, the sample ID would be the sample mode, and the KEGG ID would be the gene mode. Additionally, if you want to account for batch effects in your data, the batch variable should relate to your sample mode.\n", "- You will define two thresholds that will apply to each slab (e.g. taxon):\n", " 1. Sample threshold: each gene must be detected in at least this many samples. Genes that fall below the threshold will be removed. Default is 3.\n", " 1. Detection threshold: each sample must contain non-zero values for at least this proportion of genes. Samples that fall below the threshold will be removed. Default is 0.01 (1%).\n", " - Note that data removed due to thresholding is preserved, saved, and displayed at the end of this notebook so it can be used for troubleshooting or to refine the thresholds and normalization work flow.\n", "- If you want the normalization model to correct for batch effects, you will need to prepare a second csv file with columns corresponding to the sample mode in your data (including both sample ID and replicate), and with an additional column indicating the batch ID. For example, you might have a csv with the following headers: `['sample_id', 'replicate', 'batch_id']`. This file should include each unique combination of sample ID and replicate in your dataset, and the batch membership of each sample replicate should be indicated in the batch ID column.\n" ] }, { "cell_type": "code", "execution_count": 20, "id": "58228a6d", "metadata": {}, "outputs": [], "source": [ "# USER INPUTS -- edit these variables as needed\n", "\n", "# data inputs\n", "datapath = 'data/example-data.csv' # Enter the filepath of your input data file\n", "mode0 = 'KOfam' # Enter the column name that will correspond to the first mode of your tensor\n", "mode1 = 'phylum' # Enter the column name that will correspond to the second mode of your tensor\n", "mode2 = 'sample_name' # Enter the column name that will correspond to the third mode of your tensor\n", "rep = 'replicate' # Enter the column name that corresponds to replicate IDs\n", "data = 'counts' # Enter the column name that corresponds to your data\n", "outdir = 'data/barnacle/normalization' # Enter the filepath of the output directory where you want files saved\n", "\n", "# normalization parameters\n", "norm_mode = 'phylum' # Which mode do you want to normalize by? (enter name of mode)\n", "sample_mode = 'sample_name' # Which mode corresponds to your sample variable? (enter name of mode)\n", "sample_thold = 3 # Each {gene_mode} must be detected in what minimum number of unique {sample_mode}s? (Default is 3)\n", "gene_thold = 0.01 # Each {sample_mode} must contain what minimum proportion of nonzero {gene_mode}s? (Default is 0.01)\n", "save_data = True # Would you like to save the data output for each normalization? (Enter True/False)\n", "save_plots = True # Would you like to save the diagnostic plots for each normalization? (Enter True/False)\n", "correction = False # Would you like to correct for batch effects? (Enter True/False)\n", "metadata_path = 'data/metadata.csv' # Enter the filepath of your batch metadata file\n", "batch_id = 'batch_id' # Enter the column name that corresponds to the batch ID\n", "\n", "# check output directory exists\n", "if not os.path.exists(outdir):\n", " os.makedirs(outdir)\n", "\n", "# check data file exists\n", "if datapath and not os.path.isfile(datapath):\n", " raise Exception(f'Unable to find the file \"{datapath}\"')\n" ] }, { "cell_type": "code", "execution_count": 21, "id": "c42ba2d2-7309-431e-82ce-fcd1bf439a80", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
KOfamphylumsample_namereplicatecountssample_rep_id
0K00002BacillariophytaG3.UW.ALL.L25S1A21.00001G3.UW.ALL.L25S1_A
1K00003BacillariophytaG3.UW.ALL.L25S1A0.00000G3.UW.ALL.L25S1_A
2K00006BacillariophytaG3.UW.ALL.L25S1A3.00002G3.UW.ALL.L25S1_A
3K00010BacillariophytaG3.UW.ALL.L25S1A161.58657G3.UW.ALL.L25S1_A
4K00011BacillariophytaG3.UW.ALL.L25S1A3.00000G3.UW.ALL.L25S1_A
.....................
961825K26159PelagophyceaeG3.UW.ALL.L40S2C0.00000G3.UW.ALL.L40S2_C
961826K26163PelagophyceaeG3.UW.ALL.L40S2C1657.68445G3.UW.ALL.L40S2_C
961827K26165PelagophyceaeG3.UW.ALL.L40S2C23.19290G3.UW.ALL.L40S2_C
961828K26167PelagophyceaeG3.UW.ALL.L40S2C12.69730G3.UW.ALL.L40S2_C
961829K26171PelagophyceaeG3.UW.ALL.L40S2C3.00000G3.UW.ALL.L40S2_C
\n", "

961830 rows × 6 columns

\n", "
" ], "text/plain": [ " KOfam phylum sample_name replicate counts \\\n", "0 K00002 Bacillariophyta G3.UW.ALL.L25S1 A 21.00001 \n", "1 K00003 Bacillariophyta G3.UW.ALL.L25S1 A 0.00000 \n", "2 K00006 Bacillariophyta G3.UW.ALL.L25S1 A 3.00002 \n", "3 K00010 Bacillariophyta G3.UW.ALL.L25S1 A 161.58657 \n", "4 K00011 Bacillariophyta G3.UW.ALL.L25S1 A 3.00000 \n", "... ... ... ... ... ... \n", "961825 K26159 Pelagophyceae G3.UW.ALL.L40S2 C 0.00000 \n", "961826 K26163 Pelagophyceae G3.UW.ALL.L40S2 C 1657.68445 \n", "961827 K26165 Pelagophyceae G3.UW.ALL.L40S2 C 23.19290 \n", "961828 K26167 Pelagophyceae G3.UW.ALL.L40S2 C 12.69730 \n", "961829 K26171 Pelagophyceae G3.UW.ALL.L40S2 C 3.00000 \n", "\n", " sample_rep_id \n", "0 G3.UW.ALL.L25S1_A \n", "1 G3.UW.ALL.L25S1_A \n", "2 G3.UW.ALL.L25S1_A \n", "3 G3.UW.ALL.L25S1_A \n", "4 G3.UW.ALL.L25S1_A \n", "... ... \n", "961825 G3.UW.ALL.L40S2_C \n", "961826 G3.UW.ALL.L40S2_C \n", "961827 G3.UW.ALL.L40S2_C \n", "961828 G3.UW.ALL.L40S2_C \n", "961829 G3.UW.ALL.L40S2_C \n", "\n", "[961830 rows x 6 columns]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# check the dataframes\n", "\n", "# read in csv\n", "df = pd.read_csv(datapath)\n", "\n", "# check column names match inputs\n", "for column in [mode0, mode1, mode2, rep, data]:\n", " if column not in df.columns:\n", " raise Exception(f'Column name \"{column}\" not found in headers of file {datapath}')\n", "\n", "# define gene mode\n", "gene_mode = [m for m in [mode0, mode1, mode2] if m not in [norm_mode, sample_mode]][0]\n", "\n", "# tidy up dataframe\n", "df = df[[mode0, mode1, mode2, rep, data]]\n", "\n", "# make a unique identifier that combines sampleID and replicateID\n", "df['sample_rep_id'] = df[sample_mode].astype(str) + '_' + df[rep].astype(str)\n", "display(df)\n", "\n", "# read in metadata if using for batch correction\n", "if correction:\n", " # check data file exists\n", " if not os.path.isfile(metadata_path):\n", " raise Exception(f'Unable to find the file \"{metadata_path}\"')\n", " # read in metadata file\n", " meta_df = pd.read_csv(metadata_path)\n", " # Batch id name cannot end in \"y\" or sctransform freaks out\n", " batch_id_new = batch_id[:-1] if batch_id[-1] == 'y' else batch_id\n", " meta_df[batch_id_new] = meta_df[batch_id]\n", " batch_id = batch_id_new\n", " \n", " # check for sample, replicate, and batch columns\n", " if sample_mode not in meta_df.columns:\n", " raise Exception(f'Column name \"{sample_mode}\" not found in headers of file {metadata_path}. Columns found: {meta_df.columns}')\n", " if rep not in meta_df.columns:\n", " raise Exception(f'Column name \"{rep}\" not found in headers of file {metadata_path}. Columns found: {meta_df.columns}')\n", " if not (meta_df[[sample_mode, rep]].drop_duplicates().reset_index(drop=True) == \\\n", " df[[sample_mode, rep]].drop_duplicates().reset_index(drop=True)).all().all():\n", " raise Exception(f'Files {datapath} and {metadata_path} have different values for columns {sample_mode} and {rep}.') \n", " if batch_id not in meta_df.columns:\n", " raise Exception(f'Column name \"{batch_id}\" not found in headers of file {metadata_path}. Columns found: {meta_df.columns}')\n", "\n", " # clean up metadata dataframe\n", " meta_df['sample_rep_id'] = meta_df[sample_mode].astype(str) + '_' + meta_df[rep].astype(str)\n", " meta_df = meta_df[['sample_rep_id', sample_mode, rep, batch_id]]\n", " display(meta_df)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "dfdd23c7-c87d-42a6-b121-966fea2ce1c6", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "98d45800", "metadata": {}, "source": [ "### Run sctransform on each slab of data\n", "\n", "In this step you'll run the sctransform model on your data. If you selected it in the first step, the data and/or plots resulting from each normalization run will be saved to the designated output directory." ] }, { "cell_type": "code", "execution_count": 17, "id": "c3f66e8d-10c8-4337-8bea-bd152caf5163", "metadata": {}, "outputs": [], "source": [ "# helper functions for running normalization model\n", "\n", "# function to threshold sparsity of pandas dataframes\n", "def sparsity_thold_df(df, thold, axis=0):\n", " \"\"\"Apply a sparsity threshold to a pandas.DataFrame so that only columns or rows with \n", " greater than or equal to the threshold amount of nonzero values are retained.\n", " \n", " Parameters\n", " ----------\n", " df : pandas.DataFrame\n", " Input dataframe object.\n", " thold : {int, float}\n", " Threshold. If parameter is an int, then retained vectors must contain at least that number of \n", " nonzero values. If parameter is a float, then at least this proportion of the vector must be nonzero.\n", " axis : int\n", " Axis to which threshold is applied. \n", " \n", " Returns\n", " -------\n", " df : pandas.DataFrame\n", " Thresholded dataframe.\n", " dropped_data_df : pandas.DataFrame\n", " Data removed from the input dataframe as a result of the threshold\n", " \"\"\"\n", " if type(thold) is float:\n", " thold = math.ceil(df.shape[axis] * thold)\n", " elif type(thold) is not int:\n", " raise Exception('Parameter `thold` must be either an int or float type value.')\n", " mask = (df != 0).sum(axis).ge(thold)\n", " if axis == 0:\n", " output_df = df.loc[:, mask]\n", " dropped_df = df.loc[:, ~mask]\n", " elif axis == 1:\n", " output_df = df.loc[mask, :]\n", " dropped_df = df.loc[~mask, :]\n", " else:\n", " raise Exception('Invalid value for `axis` parameter.')\n", " return output_df, dropped_df\n", "\n", "\n", "# function to calculate 0-sensitive geometric mean\n", "def geometric_mean(vector, pseudocount=1):\n", " return np.exp(np.mean(np.log(vector + pseudocount))) - pseudocount\n", "\n", "\n", "# function to convert pandas dataframe to r matrix\n", "def pandas_dataframe_to_r_matrix(df, dtype=float):\n", " \"\"\"\n", " Function to convert pandas DataFrame objects to R matrix objects.\n", " \"\"\"\n", " if dtype is float:\n", " vector = ro.vectors.FloatVector(df.values.flatten().tolist())\n", " elif dtype is str:\n", " vector = ro.vectors.StrVector(df.values.flatten().tolist())\n", " elif dtype is int:\n", " vector = ro.vectors.FloatVector(df.values.flatten().tolist())\n", " else:\n", " raise ValueError('The dtype {} is not recognized'.format(dtype))\n", " matrix = rmatrix.Matrix(\n", " data=vector, \n", " nrow=df.shape[0], \n", " ncol=df.shape[1], \n", " byrow=True, \n", " dimnames=[df.index.to_list(), df.columns.to_list()], \n", " sparse=True\n", " )\n", " return matrix\n", " " ] }, { "cell_type": "code", "execution_count": 25, "id": "6d991bd7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Normalizing slab 1 of 8: Bacillariophyta (28 samples, 7076 genes)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "R callback write-console: vst.flavor='v2' set. Using model with fixed slope and excluding poisson genes.\n", " \n", "R callback write-console: Calculating cell attributes from input UMI matrix: log_umi\n", " \n", "R callback write-console: Total Step 1 genes: 7076\n", " \n", "R callback write-console: Total overdispersed genes: 7075\n", " \n", "R callback write-console: Excluding 1 genes from Step 1 because they are not overdispersed.\n", " \n", "R callback write-console: Calculating cell attributes from input UMI matrix: log_umi\n", " \n", "R callback write-console: Total Step 1 genes: 7076\n", " \n", "R callback write-console: Total overdispersed genes: 7075\n", " \n", "R callback write-console: Excluding 1 genes from Step 1 because they are not overdispersed.\n", " \n", "R callback write-console: Variance stabilizing transformation of count matrix of size 7076 by 28\n", " \n", "R callback write-console: Model formula is y ~ log_umi\n", " \n", "R callback write-console: Get Negative Binomial regression parameters per gene\n", " \n", "R callback write-console: Using 2000 genes, 28 cells\n", " \n", "R callback write-console: Variance stabilizing transformation of count matrix of size 7076 by 28\n", " \n", "R callback write-console: Model formula is y ~ log_umi\n", " \n", "R callback write-console: Get Negative Binomial regression parameters per gene\n", " \n", "R callback write-console: Using 2000 genes, 28 cells\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " | | 0%" ] }, { "name": "stderr", "output_type": "stream", "text": [ "R callback write-console: Error in rowAnys(x, rows, cols, value = NA, ..., useNames = useNames) : \n", " Argument 'useNames' must be either TRUE or FALSE\n", " \n" ] }, { "ename": "RRuntimeError", "evalue": "Error in rowAnys(x, rows, cols, value = NA, ..., useNames = useNames) : \n Argument 'useNames' must be either TRUE or FALSE\n", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mRRuntimeError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[25], line 53\u001b[0m\n\u001b[1;32m 48\u001b[0m result \u001b[38;5;241m=\u001b[39m sctransform\u001b[38;5;241m.\u001b[39mvst(\n\u001b[1;32m 49\u001b[0m r_slab_df, cell_attr\u001b[38;5;241m=\u001b[39mr_sample_attr_df, batch_var\u001b[38;5;241m=\u001b[39mro\u001b[38;5;241m.\u001b[39mvectors\u001b[38;5;241m.\u001b[39mStrVector([batch_id]), \n\u001b[1;32m 50\u001b[0m min_cells\u001b[38;5;241m=\u001b[39msample_thold, vst_flavor\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mv2\u001b[39m\u001b[38;5;124m'\u001b[39m, verbosity\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m2\u001b[39m\n\u001b[1;32m 51\u001b[0m )\n\u001b[1;32m 52\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m---> 53\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[43msctransform\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvst\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[43m \u001b[49m\u001b[43mr_slab_df\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmin_cells\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msample_thold\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvst_flavor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mv2\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbosity\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\n\u001b[1;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 57\u001b[0m \u001b[38;5;66;03m# convert residuals result to a dataframe\u001b[39;00m\n\u001b[1;32m 58\u001b[0m result_df \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mDataFrame(\n\u001b[1;32m 59\u001b[0m np\u001b[38;5;241m.\u001b[39masarray(result[\u001b[38;5;241m0\u001b[39m]), \n\u001b[1;32m 60\u001b[0m index\u001b[38;5;241m=\u001b[39mslab_df\u001b[38;5;241m.\u001b[39mindex, \n\u001b[1;32m 61\u001b[0m columns\u001b[38;5;241m=\u001b[39mslab_df\u001b[38;5;241m.\u001b[39mcolumns\n\u001b[1;32m 62\u001b[0m )\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/robjects/functions.py:208\u001b[0m, in \u001b[0;36mSignatureTranslatedFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 206\u001b[0m v \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(k)\n\u001b[1;32m 207\u001b[0m kwargs[r_k] \u001b[38;5;241m=\u001b[39m v\n\u001b[0;32m--> 208\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m (\u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mSignatureTranslatedFunction\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 209\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__call__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m)\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/robjects/functions.py:131\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 129\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 130\u001b[0m new_kwargs[k] \u001b[38;5;241m=\u001b[39m cv\u001b[38;5;241m.\u001b[39mpy2rpy(v)\n\u001b[0;32m--> 131\u001b[0m res \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mFunction\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__call__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mnew_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mnew_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 132\u001b[0m res \u001b[38;5;241m=\u001b[39m cv\u001b[38;5;241m.\u001b[39mrpy2py(res)\n\u001b[1;32m 133\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m res\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface_lib/conversion.py:46\u001b[0m, in \u001b[0;36m_cdata_res_to_rinterface.._\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 45\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m---> 46\u001b[0m cdata \u001b[38;5;241m=\u001b[39m \u001b[43mfunction\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 47\u001b[0m \u001b[38;5;66;03m# TODO: test cdata is of the expected CType\u001b[39;00m\n\u001b[1;32m 48\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m _cdata_to_rinterface(cdata)\n", "File \u001b[0;32m~/miniconda3/envs/barnacle311/lib/python3.11/site-packages/rpy2/rinterface/__init__.py:926\u001b[0m, in \u001b[0;36mSexpClosure.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 919\u001b[0m res \u001b[38;5;241m=\u001b[39m rmemory\u001b[38;5;241m.\u001b[39mprotect(\n\u001b[1;32m 920\u001b[0m openrlib\u001b[38;5;241m.\u001b[39mrlib\u001b[38;5;241m.\u001b[39mR_tryEval(\n\u001b[1;32m 921\u001b[0m call_r,\n\u001b[1;32m 922\u001b[0m call_context\u001b[38;5;241m.\u001b[39m__sexp__\u001b[38;5;241m.\u001b[39m_cdata,\n\u001b[1;32m 923\u001b[0m error_occured)\n\u001b[1;32m 924\u001b[0m )\n\u001b[1;32m 925\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m error_occured[\u001b[38;5;241m0\u001b[39m]:\n\u001b[0;32m--> 926\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m embedded\u001b[38;5;241m.\u001b[39mRRuntimeError(_rinterface\u001b[38;5;241m.\u001b[39m_geterrmessage())\n\u001b[1;32m 927\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m res\n", "\u001b[0;31mRRuntimeError\u001b[0m: Error in rowAnys(x, rows, cols, value = NA, ..., useNames = useNames) : \n Argument 'useNames' must be either TRUE or FALSE\n" ] } ], "source": [ "# run the model on each slab\n", "\n", "# initialize residuals dataframe\n", "residuals_df = pd.DataFrame()\n", "\n", "# keep track of filtered data\n", "dropped_data_df = pd.DataFrame()\n", "\n", "# iterate through slabs\n", "slab_ids = df[norm_mode].unique()\n", "for i, slab_id in enumerate(slab_ids):\n", " # separate out data\n", " slab_df = df[df[norm_mode].eq(slab_id)].pivot(index=gene_mode, columns=['sample_rep_id'], values=data).fillna(0)\n", " # apply sample threshold to filter out low-prevalence genes\n", " slab_df, drop_df = sparsity_thold_df(slab_df, sample_thold, axis=1)\n", " drop_df = drop_df.melt(value_name=data, ignore_index=False).reset_index()\n", " drop_df[norm_mode] = slab_id\n", " drop_df['drop_reason'] = f'{gene_mode} detected in fewer than {sample_thold} {sample_mode}s'\n", " dropped_data_df = pd.concat([dropped_data_df, drop_df])\n", " # apply gene threshold to filter out samples with low detection\n", " slab_df, drop_df = sparsity_thold_df(slab_df, gene_thold, axis=0)\n", " drop_df = drop_df.melt(value_name=data, ignore_index=False).reset_index()\n", " drop_df[norm_mode] = slab_id\n", " drop_df['drop_reason'] = f'{sample_mode} contains fewer than {math.ceil(slab_df.shape[0] * gene_thold)} nonzero {gene_mode}s'\n", " dropped_data_df = pd.concat([dropped_data_df, drop_df])\n", " # check for very small slabs\n", " if (slab_df.shape[0] < 10) or (slab_df.shape[1] < sample_thold):\n", " print(f'Skipping slab {i+1} of {len(slab_ids)}: {slab_id} ({slab_df.shape[1]} samples, {slab_df.shape[0]} genes)', flush=True)\n", " print('\\tLimited nonzero data in this slab undermines the reliability of normalization with sctransform.', flush=True)\n", " drop_df = slab_df.melt(value_name=data, ignore_index=False).reset_index()\n", " drop_df[norm_mode] = slab_id\n", " drop_df['drop_reason'] = f'{norm_mode} encompassed fewer than 10 {gene_mode}s or fewer than {sample_thold} {sample_mode}s'\n", " dropped_data_df = pd.concat([dropped_data_df, drop_df])\n", " continue\n", " else:\n", " print(f'Normalizing slab {i+1} of {len(slab_ids)}: {slab_id} ({slab_df.shape[1]} samples, {slab_df.shape[0]} genes)', flush=True)\n", "\n", " # make r version of slab dataframe\n", " r_slab_df = pandas_dataframe_to_r_matrix(slab_df)\n", " # pull out batch information\n", " if correction:\n", " sample_attr_df = meta_df.set_index('sample_rep_id').loc[slab_df.columns, [sample_mode, rep, batch_id]]\n", " with localconverter(ro.default_converter + pandas2ri.converter):\n", " r_sample_attr_df = ro.conversion.py2rpy(sample_attr_df)\n", "\n", " # fit vst normalization model\n", " if correction:\n", " result = sctransform.vst(\n", " r_slab_df, cell_attr=r_sample_attr_df, batch_var=ro.vectors.StrVector([batch_id]), \n", " min_cells=sample_thold, vst_flavor='v2', verbosity=2\n", " )\n", " else:\n", " result = sctransform.vst(\n", " r_slab_df, min_cells=sample_thold, vst_flavor='v2', verbosity=2\n", " )\n", "\n", " # convert residuals result to a dataframe\n", " result_df = pd.DataFrame(\n", " np.asarray(result[0]), \n", " index=slab_df.index, \n", " columns=slab_df.columns\n", " )\n", "\n", " # prepare to save requested outputs\n", " if save_data or save_plots:\n", " # make unique output directory per slab\n", " dir_path = f'{outdir}/{slab_id}'\n", " if not os.path.isdir(dir_path):\n", " os.makedirs(dir_path)\n", " # calculate residuals\n", " residual_var = result_df.var(axis=1)\n", "\n", " # save data if requested\n", " if save_data:\n", " # save csv of residuals\n", " result_df.to_csv(f'{dir_path}/residuals_{slab_id}.csv')\n", " # save csv of residual variances\n", " res_var_df = residual_var.reset_index().rename(columns={0:'residual_variance'})\n", " res_var_df = res_var_df.sort_values('residual_variance', ascending=False).reset_index()\n", " res_var_df.to_csv(f'{dir_path}/residual_variances_{slab_id}.csv')\n", "\n", " # save plots if requested\n", " if save_plots:\n", " # plots of model parameters\n", " plots = sctransform.plot_model_pars(result, show_theta=True)\n", " img = image_png(plots)\n", " with open(f'{dir_path}/parameters_{slab_id}.png', 'wb') as png:\n", " png.write(img.data)\n", " # plot of high variance genes\n", " means = slab_df.apply(geometric_mean, axis=1)\n", " plt.figure(figsize=(10, 4))\n", " sns.scatterplot(x=means, y=residual_var, alpha=0.1);\n", " plt.xlabel(f'geometric mean of {gene_mode} {data}')\n", " plt.xscale('log')\n", " plt.ylabel('residual variance')\n", " plt.title(f'normalized residual variance\\nvs. mean {gene_mode} {data} ({slab_id})')\n", " plt.savefig(f'{dir_path}/residual_variances_{slab_id}.png')\n", " plt.show()\n", "\n", " # concatenate result with other residuals\n", " result_df = result_df.melt(value_name='residual', ignore_index=False).reset_index()\n", " result_df[norm_mode] = slab_id\n", " residuals_df = pd.concat([residuals_df, result_df])\n", " " ] }, { "cell_type": "code", "execution_count": 7, "id": "3e5fde11", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
KOfamphylumsample_namereplicatecountsresidual
0K00001BacillariophytaG3.UW.ALL.L25S1A0.00000-9.222961e-13
1K00002BacillariophytaG3.UW.ALL.L25S1A21.000012.092099e+00
2K00003BacillariophytaG3.UW.ALL.L25S1A0.00000-1.128763e+00
3K00004BacillariophytaG3.UW.ALL.L25S1A0.00000-1.635693e-02
4K00006BacillariophytaG3.UW.ALL.L25S1A3.00002-6.474356e-01
.....................
1284467K26159PelagophyceaeG3.UW.ALL.L40S2C0.00000-1.360729e-01
1284468K26163PelagophyceaeG3.UW.ALL.L40S2C1657.684451.537219e+00
1284469K26165PelagophyceaeG3.UW.ALL.L40S2C23.192906.104757e-01
1284470K26167PelagophyceaeG3.UW.ALL.L40S2C12.697308.077639e-01
1284471K26171PelagophyceaeG3.UW.ALL.L40S2C3.00000-3.022025e-01
\n", "

1284472 rows × 6 columns

\n", "
" ], "text/plain": [ " KOfam phylum sample_name replicate counts \\\n", "0 K00001 Bacillariophyta G3.UW.ALL.L25S1 A 0.00000 \n", "1 K00002 Bacillariophyta G3.UW.ALL.L25S1 A 21.00001 \n", "2 K00003 Bacillariophyta G3.UW.ALL.L25S1 A 0.00000 \n", "3 K00004 Bacillariophyta G3.UW.ALL.L25S1 A 0.00000 \n", "4 K00006 Bacillariophyta G3.UW.ALL.L25S1 A 3.00002 \n", "... ... ... ... ... ... \n", "1284467 K26159 Pelagophyceae G3.UW.ALL.L40S2 C 0.00000 \n", "1284468 K26163 Pelagophyceae G3.UW.ALL.L40S2 C 1657.68445 \n", "1284469 K26165 Pelagophyceae G3.UW.ALL.L40S2 C 23.19290 \n", "1284470 K26167 Pelagophyceae G3.UW.ALL.L40S2 C 12.69730 \n", "1284471 K26171 Pelagophyceae G3.UW.ALL.L40S2 C 3.00000 \n", "\n", " residual \n", "0 -9.222961e-13 \n", "1 2.092099e+00 \n", "2 -1.128763e+00 \n", "3 -1.635693e-02 \n", "4 -6.474356e-01 \n", "... ... \n", "1284467 -1.360729e-01 \n", "1284468 1.537219e+00 \n", "1284469 6.104757e-01 \n", "1284470 8.077639e-01 \n", "1284471 -3.022025e-01 \n", "\n", "[1284472 rows x 6 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# save normalized data as a csv\n", "\n", "# add back sample information\n", "residuals_df = pd.merge(residuals_df, df[['sample_rep_id', sample_mode, rep]].drop_duplicates(), on='sample_rep_id', how='left')\n", "\n", "# add in raw count data\n", "residuals_df = pd.merge(residuals_df, df, on=['sample_rep_id', mode0, mode1, mode2, rep], how='left').fillna(0)\n", "\n", "# tidy up dataframe\n", "residuals_df = residuals_df[[mode0, mode1, mode2, rep, data, 'residual']]\n", "\n", "# save output\n", "residuals_df.to_csv(f'{outdir}/normalized-residuals.csv')\n", "\n", "residuals_df\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "012df29b-3d4d-410e-a2c6-ec802400b1ba", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "drop_reason\n", "KOfam detected in fewer than 3 sample_names 3017\n", "Name: count, dtype: int64\n", "KOfam\n", "K00015 6\n", "K25390 5\n", "K15462 5\n", "K16422 5\n", "K10638 4\n", " ..\n", "K19222 1\n", "K22089 1\n", "K01368 1\n", "K03025 1\n", "K17064 1\n", "Name: count, Length: 1757, dtype: int64\n", "phylum\n", "Haptophyta 494\n", "Bacillariophyta 436\n", "Pelagophyceae 428\n", "Dictyochophyceae 414\n", "Chlorophyta 383\n", "Bolidophyceae 312\n", "Chrysophyceae 309\n", "Ciliophora 241\n", "Name: count, dtype: int64\n", "sample_name\n", "G3.UW.ALL.L40S1 962\n", "G3.UW.ALL.L40S2 321\n", "G3.UW.ALL.L37S1 273\n", "G3.UW.ALL.L35S1 270\n", "G3.UW.ALL.L38S1 250\n", "G3.UW.ALL.L35S2 199\n", "G3.UW.ALL.L32S3 168\n", "G3.UW.ALL.L25S1 159\n", "G3.UW.ALL.L32S1 152\n", "G3.UW.ALL.L29S1 139\n", "G3.UW.ALL.L31S2 124\n", "Name: count, dtype: int64\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
KOfamphylumsample_namereplicatecountsdrop_reason
0K04261BacillariophytaG3.UW.ALL.L25S1A2.00000KOfam detected in fewer than 3 sample_names
1K10720BacillariophytaG3.UW.ALL.L25S1A1.00000KOfam detected in fewer than 3 sample_names
2K17353BacillariophytaG3.UW.ALL.L25S1A6.35636KOfam detected in fewer than 3 sample_names
3K20847BacillariophytaG3.UW.ALL.L25S1A1.00000KOfam detected in fewer than 3 sample_names
4K22619BacillariophytaG3.UW.ALL.L25S1A4.74145KOfam detected in fewer than 3 sample_names
.....................
3012K16197PelagophyceaeG3.UW.ALL.L40S2C4.00000KOfam detected in fewer than 3 sample_names
3013K16581PelagophyceaeG3.UW.ALL.L40S2C10.95080KOfam detected in fewer than 3 sample_names
3014K16634PelagophyceaeG3.UW.ALL.L40S2C6.00000KOfam detected in fewer than 3 sample_names
3015K17064PelagophyceaeG3.UW.ALL.L40S2C1.33754KOfam detected in fewer than 3 sample_names
3016K22679PelagophyceaeG3.UW.ALL.L40S2C5.06391KOfam detected in fewer than 3 sample_names
\n", "

3017 rows × 6 columns

\n", "
" ], "text/plain": [ " KOfam phylum sample_name replicate counts \\\n", "0 K04261 Bacillariophyta G3.UW.ALL.L25S1 A 2.00000 \n", "1 K10720 Bacillariophyta G3.UW.ALL.L25S1 A 1.00000 \n", "2 K17353 Bacillariophyta G3.UW.ALL.L25S1 A 6.35636 \n", "3 K20847 Bacillariophyta G3.UW.ALL.L25S1 A 1.00000 \n", "4 K22619 Bacillariophyta G3.UW.ALL.L25S1 A 4.74145 \n", "... ... ... ... ... ... \n", "3012 K16197 Pelagophyceae G3.UW.ALL.L40S2 C 4.00000 \n", "3013 K16581 Pelagophyceae G3.UW.ALL.L40S2 C 10.95080 \n", "3014 K16634 Pelagophyceae G3.UW.ALL.L40S2 C 6.00000 \n", "3015 K17064 Pelagophyceae G3.UW.ALL.L40S2 C 1.33754 \n", "3016 K22679 Pelagophyceae G3.UW.ALL.L40S2 C 5.06391 \n", "\n", " drop_reason \n", "0 KOfam detected in fewer than 3 sample_names \n", "1 KOfam detected in fewer than 3 sample_names \n", "2 KOfam detected in fewer than 3 sample_names \n", "3 KOfam detected in fewer than 3 sample_names \n", "4 KOfam detected in fewer than 3 sample_names \n", "... ... \n", "3012 KOfam detected in fewer than 3 sample_names \n", "3013 KOfam detected in fewer than 3 sample_names \n", "3014 KOfam detected in fewer than 3 sample_names \n", "3015 KOfam detected in fewer than 3 sample_names \n", "3016 KOfam detected in fewer than 3 sample_names \n", "\n", "[3017 rows x 6 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# examine data that was removed during normalization process\n", "\n", "# remove zero values\n", "dropped_data_df = dropped_data_df[dropped_data_df[data] != 0.0]\n", "\n", "# add back sample information\n", "dropped_data_df = pd.merge(dropped_data_df, df[['sample_rep_id', sample_mode, rep]].drop_duplicates(), on='sample_rep_id', how='left')\n", "\n", "# tidy up dataframe\n", "dropped_data_df = dropped_data_df[[mode0, mode1, mode2, rep, data, 'drop_reason']]\n", "\n", "# show some summary statistics\n", "for variable in ['drop_reason', mode0, mode1, mode2]:\n", " print(dropped_data_df[variable].value_counts())\n", "\n", "# save drop data\n", "dropped_data_df.to_csv(f'{outdir}/removed-data.csv') \n", "\n", "dropped_data_df\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ac06d6d9-9513-4a6c-be82-0ec7626028ce", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "barnacle311", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 5 }