Problem with dataset size

[rubencmu]

Hi, can I get help with this question that was asked in the CCD help desk. The user is working with the py-causal implementation.
Thanks.
I paste it below:

---

Also, I have a large (8GB) csv file that is able to eventually break your code:

Exception in thread "Thread-0" java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:869)
at edu.cmu.tetrad.io.AbstractContinuousDataReader.extractVariables(AbstractContinuousDataReader.java:75)
at edu.cmu.tetrad.io.AbstractContinuousDataReader.analyzeData(AbstractContinuousDataReader.java:59)
at edu.cmu.tetrad.io.TabularContinuousDataReader.readInData(TabularContinuousDataReader.java:62)

[chirayukong]

@jdramsey @kvb2univpitt It might be related to the DataReader module.

[jdramsey]

This isn't my code, but that's this line:

    try (FileChannel fc = new RandomAccessFile(dataFile.toFile(), "r").getChannel()) {
        MappedByteBuffer buffer = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());

The exception is in the second line.

[kvb2univpitt]

I'll take a look at the data reader. I just need to find 8GB or larger dataset to test on.

[espinoj]

@rubencmu can you please share the file with @kvb2univpitt so he can replicate/debug

[rubencmu]

I will ask the user to share the location of the data. I will share it as soon as I get a response.
@kvb2univpitt @espinoj

[rubencmu]

@espinoj @kvb2univpitt Here is the answer from the user:

---

Yeah! It's the Higgs Boson dataset. I managed to contort things so it parsed. Deciding if I want to try to fix Javabridge (even if just with multiprocessing).

https://archive.ics.uci.edu/ml/datasets/HIGGS

---

[kvb2univpitt]

Sorry for the late response. There's a bug with reading large file via memory-mapped. The size of the region to be mapped cannot be greater than the max integer, even though the size parameter in the map method is a long type.

public abstract MappedByteBuffer map(MapMode mode, long position, long size)

This will be fixed in the new data reader.