New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prototype for storing single-cell data #1020

Open

arteymix wants to merge 496 commits into development from feature-single-cell

Member

arteymix commented Feb 5, 2024 •

edited

Loading

TODO

add a CLI for loading/reloading single cell data and cell type assignments
extend the GEO loader to detect MEX and other supported formats and apply the appropriate strategy for loading vectors
Add support for HDF5-based single-cell data formats #1039
support saving single cell data to disk (I already added a SingleCellExpresionDataMatrix, we need to finish the work and write it to file). I think MEX is a pretty decent output format for this.

REST API

review which fields should be exposed on the REST API for filtering purposes
add aliases in the REST API to refer to the preferred single cell dimension and cell assignment

arteymix force-pushed the feature-single-cell branch 6 times, most recently from 9222a95 to 788a61b Compare

February 7, 2024 20:13

arteymix commented

View reviewed changes

gemma-core/src/test/java/ubic/gemma/persistence/util/ListUtilsTest.java Outdated Show resolved Hide resolved

arteymix force-pushed the feature-single-cell branch 2 times, most recently from f04ae2f to d791771 Compare

February 7, 2024 20:23

arteymix commented

View reviewed changes

...va/ubic/gemma/persistence/service/expression/experiment/ExpressionExperimentServiceImpl.java Outdated Show resolved Hide resolved

arteymix commented

View reviewed changes

.../main/java/ubic/gemma/persistence/service/expression/experiment/ExpressionExperimentDao.java Show resolved Hide resolved

arteymix commented

View reviewed changes

gemma-core/src/main/java/ubic/gemma/model/expression/bioAssayData/DataVector.java Outdated Show resolved Hide resolved

arteymix commented

View reviewed changes

...-core/src/main/resources/ubic/gemma/model/expression/designElement/CompositeSequence.hbm.xml Outdated Show resolved Hide resolved

arteymix force-pushed the feature-single-cell branch from b60a8de to 0ce142a Compare

February 8, 2024 00:23

This was linked to issues Feb 10, 2024

Store original cell-by-gene matrix #1012

Closed

Identify the input formats for single-cell data we want to support in Gemma #1028

Closed

arteymix force-pushed the feature-single-cell branch from 80c6409 to e804d92 Compare

February 13, 2024 03:40

This was linked to issues Feb 16, 2024

Import additional characteristics from Seurat Disk and AnnData formats #1037

Closed

Use "cell type assignment" instead of "cell type labelling" #1038

Closed

arteymix force-pushed the feature-single-cell branch 4 times, most recently from b2c8a8b to 6a993b7 Compare

February 19, 2024 20:47

arteymix added the single cell label

arteymix self-assigned this

arteymix force-pushed the feature-single-cell branch from b7d4810 to 7c29995 Compare

February 21, 2024 23:26

arteymix linked an issue

that may be closed by this pull request

Load single-cell data from GEO #1029

Closed

3 tasks

arteymix force-pushed the development branch from 5bfbd30 to c2e50b6 Compare

June 8, 2024 06:24

Member Author

arteymix commented Jun 11, 2024

I'm in the process of merging the dev branch to get this work up-to-date.

arteymix commented

View reviewed changes

gemma-core/src/main/resources/ubic/gemma/model/analysis/Investigation.hbm.xml Outdated

@@ @@ -130,6 +130,13 @@ @@
               					<!-- cannot be non-null because subsets and generic experiments don't have curation details -->
               					<column name="CURATION_DETAILS_FK" not-null="false" sql-type="BIGINT" unique="true"/>
               				</many-to-one>
+              				<set name="singleCellExpressionDataVectors" lazy="true" fetch="select" inverse="true"
+              					 cascade="all-delete-orphan">

Member Author

arteymix Jun 12, 2024

We should remove the -delete-orphan and manage vectors the same way we do for raw and processed ones.

Member Author

arteymix Jun 12, 2024

That include bulk insertion, removal, etc.

arteymix added 22 commits

February 28, 2025 10:54


          Handle 'null' and other missing value indicators in GEO data

cadfd21

Those values should be handled gracefully and without producing a
warning.

Move conversion logic back in GeoConverterImpl since this is meant to be
tailored to data encountered in GEO.


          Bump pavlab-starter-parent to 1.2.19-SNAPSHOT

4b035cf


          Parse present/absent indicators

133259a


          Replace all usages of commons-lang with commons-lang3 or equivalent

14f14cd


          Eliminate FeedReaderController and rome-fetcher

26f10ca


          Update dependencies

fa16931

Enforce dependency convergence now that it has been achieved.

Remove unused jboss-3jb3x dependency


          Remove common prefix when when displaying subset names

a1b0831


          Make sure we use DoubleArrayList.size() or trimToSize() when accessin…

6d0621f

…g elements()

A DAL can have more elements than actual values, so its length must
always be taken from size(), not elements().length.

This is only problematic if the array was created by calling add().


          Add support for all byte array types and add a charset parameter to C…

ae2ffd3

…ompressedStringListType


          Replace sizeWithoutMissingValues usage in ComBat with more efficient …

29e606e

…loop


          Downgrade jackson to 2.18.2 to satisfy swagger

24f0502


          Add missing kotlin-stdlib-jdk8 dependency for slack

378b382


          cli: Fix parsing of --format option for getSingleCellDataMatrix

daee693


          Add options to deleteSingleCellData to remove CTAs and CLCs

fef485c


          Make single-cell sparsity heatmaps render as placeholders if there is…

deea578

… no processed or preferred vectors


          Exclude cell type factors from the experimental design overview


          Add a flag to ignore samples with incomplete/missing single-cell data

b325590

Fix #1337


          Add a test for empty MEX datasets

80d1c8a

As we can now ignore missing or incomplete samples, there is a
possibility that we import an empty MEX dataset.


          Correctly apply overrides when replacing single-cell data

df1f310

When data is replaced, the preferred flag is not honored.

Re-apply QT overrides when a QT is found by name.


          Implement file-level locking for writing expression data

1eeb4f8


          Add more tests for file locks

0871e38


          Merge pull request #1338 from PavlidisLab/feature-file-locking

d9827de

Implement file-level locking for expression data files

arteymix marked this pull request as ready for review

March 8, 2025 23:07

arteymix and others added 7 commits

March 8, 2025 16:11


          Reuse the same java classpath and binary when running a subprocess

1dcbe2c


          cli: Add a CliContext and support for aliases and improve testing

3a3c85f

The CLI context allow for mocking System-related utilities and produce
helpful error messages in tests.


          Add FileLockManager to coordinate file-level locking

59e4f1a


          Add a CLI to lock a data or metadata file

0cd6e2b

This can be used in scripts to manipulate data files safely.


          add PMID to citations. possibly fixing #1283

8bbd0fc


          exclude pubpeer comments from the css

90984b1


          Merge pull request #1339 from PavlidisLab/feature-pubpeer-visibility

eb05653

Allows pubpeer to see papers linked in experiment pages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment