Performance improvements for long reads

[mwiewior]

Optimize calculatePositionInReadSeq function to keep state and avoid reprocessing for each position.

[mwiewior]

BASELINE
=========

Time taken: 120897 ms
Timings
+-------------------------------------------------------------------------+---------------+---------------+-------------+---------+---------------+---------------+---------------+
|                                 Metric                                  | Worker Total  | Driver Total  | Driver Only |  Count  |     Mean      |      Min      |      Max      |
+-------------------------------------------------------------------------+---------------+---------------+-------------+---------+---------------+---------------+---------------+
| └─ ContigAggrTimer from PileupMethods                                   |             - |     6.18 secs |    33.12 ms |       1 |     6.18 secs |     6.18 secs |     6.18 secs |
|     ├─ InitContigLengthsTimer from PileupMethods                        |             - |      6.1 secs |    19.46 ms |       1 |      6.1 secs |      6.1 secs |      6.1 secs |
|     │   └─ first at AlignmentsRDD.scala:35                              |             - |     6.08 secs |           - |       1 |     6.08 secs |     6.08 secs |     6.08 secs |
|     └─ mapPartitions at AlignmentsRDD.scala:37                          |             - |      66.36 ms |           - |       1 |      66.36 ms |      66.36 ms |      66.36 ms |
|         └─ function call                                                | 1 min 45 secs |             - |           - |       6 |     17.6 secs |     5.38 secs |    25.09 secs |
|             └─ MapPartitionTimer from PileupMethods                     | 1 min 45 secs |             - |           - |       6 |     17.6 secs |     5.38 secs |    25.09 secs |
|                 ├─ AnalyzeReadsTimer from Read                          | 1 min 43 secs |             - |           - |   14272 |       7.28 ms |       2.98 µs |      3.6 secs |
|                 │   └─ AnalyzeReadscalculatePositionInReadSeq from Read |    55.37 secs |             - |           - | 5447228 |      10.16 µs |        115 ns |      23.23 ms |
|                 ├─ HandleFirstContingTimer from PileupMethods           |      60.84 ms |             - |           - |       6 |      10.14 ms |       3.12 ms |      32.87 ms |
|                 │   └─ AggMapLookupTimer from PileupMethods             |      91.76 µs |             - |           - |       6 |      15.29 µs |        8.6 µs |      34.58 µs |
|                 ├─ PrepareOutupTimer from PileupMethods                 |      51.07 ms |             - |           - |       6 |       8.51 ms |       6.78 ms |       11.4 ms |
|                 ├─ BAMReadTimer from PileupMethods                      |      11.96 ms |             - |           - |   14272 |        837 ns |        143 ns |     120.46 µs |
|                 └─ DQTimerTimer from PileupMethods                      |       5.09 ms |             - |           - |   14272 |        356 ns |        124 ns |      87.46 µs |
| └─ AccumulatorTimer from PileupMethods                                  |             - | 1 min 46 secs |     3.66 ms |       1 | 1 min 46 secs | 1 min 46 secs | 1 min 46 secs |
|     ├─ AccumulatorAllocTimer from PileupMethods                         |             - |      810.7 µs |    810.7 µs |       1 |      810.7 µs |      810.7 µs |      810.7 µs |
|     ├─ AccumulatorRegisterTimer from PileupMethods                      |             - |      47.76 µs |    47.76 µs |       1 |      47.76 µs |      47.76 µs |      47.76 µs |
|     └─ foreach at ContigAggregateRDD.scala:36                           |             - | 1 min 46 secs |           - |       1 | 1 min 46 secs | 1 min 46 secs | 1 min 46 secs |
|         └─ function call                                                |      71.83 ms |             - |           - |       6 |      11.97 ms |       3.64 ms |      24.55 ms |
|             └─ AccumulatorNestedTimer from PileupMethods                |      71.28 ms |             - |           - |       6 |      11.88 ms |       3.61 ms |       24.2 ms |
|                 ├─ PileupUpdateCreationTimer from PileupMethods         |      70.22 ms |             - |           - |       6 |       11.7 ms |       3.57 ms |      23.46 ms |
|                 │   ├─ TailAltsTimer from PileupMethods                 |      52.89 ms |             - |           - |       6 |       8.81 ms |       2.96 ms |      12.93 ms |
|                 │   ├─ TailCovTimer from PileupMethods                  |      11.46 ms |             - |           - |       6 |       1.91 ms |     439.45 µs |       6.75 ms |
|                 │   └─ TailEdgeTimer from PileupMethods                 |     371.92 µs |             - |           - |       6 |      61.99 µs |       4.75 µs |     313.94 µs |
|                 └─ AcumulatorAddTimer from PileupMethods                |      97.93 µs |             - |           - |       6 |      16.32 µs |       7.49 µs |       49.2 µs |
| └─ BroadcastTimer from PileupMethods                                    |             - |     899.59 ms |   899.59 ms |       1 |     899.59 ms |     899.59 ms |     899.59 ms |
| └─ AdjustedEventsTimer from PileupMethods                               |             - |       7.48 ms |        1 ms |       1 |       7.48 ms |       7.48 ms |       7.48 ms |
|     └─ map at ContigAggregateRDD.scala:46                               |             - |       6.47 ms |           - |       1 |       6.47 ms |       6.47 ms |       6.47 ms |
|         └─ function call                                                |     1.32 secs |             - |           - |       6 |     220.68 ms |       1.23 ms |     300.63 ms |
|             ├─ CalculateAltsTimer from ContigAggregate                  |     1.32 secs |             - |           - |       6 |     219.57 ms |      15.42 µs |     300.13 ms |
|             ├─ CalculateEventsTimer from ContigAggregate                |       3.92 ms |             - |           - |       6 |      654.3 µs |      50.24 µs |       2.22 ms |
|             ├─ ShrinkArrayTimer from PileupMethods                      |     120.99 µs |             - |           - |       6 |      20.16 µs |      10.53 µs |       53.2 µs |
|             └─ CalculateQualsTimer from ContigAggregate                 |     114.84 µs |             - |           - |       6 |      19.14 µs |      14.72 µs |      21.95 µs |
| └─ EventsToPileupTimer from PileupMethods                               |             - |      13.85 ms |     1.73 ms |       1 |      13.85 ms |      13.85 ms |      13.85 ms |
|     ├─ mapPartitions at ContigAggregateRDD.scala:51                     |             - |       7.05 ms |           - |       1 |       7.05 ms |       7.05 ms |       7.05 ms |
|     │   └─ function call                                                |     206.87 ms |             - |           - |       6 |      34.48 ms |       25.2 ms |      61.88 ms |
|     └─ flatMap at ContigAggregateRDD.scala:100                          |             - |       5.07 ms |           - |       1 |       5.07 ms |       5.07 ms |       5.07 ms |
|         └─ function call                                                |     150.74 µs |             - |           - |       6 |      25.12 µs |      14.71 µs |      71.28 µs |
+-------------------------------------------------------------------------+---------------+---------------+-------------+---------+---------------+---------------+---------------+

[mwiewior]

Current state

Time taken: 23175 ms
Timings
+-----------------------------------------------------------------------------+--------------+--------------+-------------+---------+-----------+-----------+-----------+
|                                   Metric                                    | Worker Total | Driver Total | Driver Only |  Count  |   Mean    |    Min    |    Max    |
+-----------------------------------------------------------------------------+--------------+--------------+-------------+---------+-----------+-----------+-----------+
| └─ ContigAggrTimer from PileupMethods                                       |            - |    6.06 secs |    32.91 ms |       1 | 6.06 secs | 6.06 secs | 6.06 secs |
|     ├─ InitContigLengthsTimer from PileupMethods                            |            - |       6 secs |    22.58 ms |       1 |    6 secs |    6 secs |    6 secs |
|     │   └─ first at AlignmentsRDD.scala:35                                  |            - |    5.98 secs |           - |       1 | 5.98 secs | 5.98 secs | 5.98 secs |
|     └─ mapPartitions at AlignmentsRDD.scala:37                              |            - |     46.91 ms |           - |       1 |  46.91 ms |  46.91 ms |  46.91 ms |
|         └─ function call                                                    |    7.86 secs |            - |           - |       6 | 1.31 secs | 519.95 ms | 1.94 secs |
|             └─ MapPartitionTimer from PileupMethods                         |    7.86 secs |            - |           - |       6 | 1.31 secs |  519.9 ms | 1.94 secs |
|                 ├─ AnalyzeReadsTimer from Read                              |    5.86 secs |            - |           - |   14272 | 410.49 µs |   2.81 µs |  69.65 ms |
|                 │   └─ AnalyzeReadsCalculateAltsLoopTimer from Read         |    2.87 secs |            - |           - |   14272 | 201.27 µs |    308 ns |  69.06 ms |
|                 │       └─ AnalyzeReadscalculatePositionInReadSeq from Read |    468.76 ms |            - |           - | 5447228 |     86 ns |     28 ns | 243.32 µs |
|                 ├─ PrepareOutupTimer from PileupMethods                     |     66.97 ms |            - |           - |       6 |  11.16 ms |   8.41 ms |  14.84 ms |
|                 ├─ HandleFirstContingTimer from PileupMethods               |     52.69 ms |            - |           - |       6 |   8.78 ms |   3.06 ms |  24.07 ms |
|                 │   └─ AggMapLookupTimer from PileupMethods                 |    101.94 µs |            - |           - |       6 |  16.99 µs |   8.61 µs |  45.67 µs |
|                 ├─ BAMReadTimer from PileupMethods                          |      4.79 ms |            - |           - |   14272 |    335 ns |     83 ns |  81.35 µs |
|                 └─ DQTimerTimer from PileupMethods                          |      3.42 ms |            - |           - |   14272 |    239 ns |     73 ns |  97.55 µs |
| └─ AccumulatorTimer from PileupMethods                                      |            - |    8.91 secs |     3.76 ms |       1 | 8.91 secs | 8.91 secs | 8.91 secs |
|     ├─ AccumulatorAllocTimer from PileupMethods                             |            - |    874.42 µs |   874.42 µs |       1 | 874.42 µs | 874.42 µs | 874.42 µs |
|     ├─ AccumulatorRegisterTimer from PileupMethods                          |            - |     60.79 µs |    60.79 µs |       1 |  60.79 µs |  60.79 µs |  60.79 µs |
|     └─ foreach at ContigAggregateRDD.scala:36                               |            - |     8.9 secs |           - |       1 |  8.9 secs |  8.9 secs |  8.9 secs |
|         └─ function call                                                    |     78.03 ms |            - |           - |       6 |     13 ms |   4.04 ms |   24.4 ms |
|             └─ AccumulatorNestedTimer from PileupMethods                    |     77.43 ms |            - |           - |       6 |   12.9 ms |      4 ms |  24.03 ms |
|                 ├─ PileupUpdateCreationTimer from PileupMethods             |     76.42 ms |            - |           - |       6 |  12.74 ms |   3.95 ms |  23.39 ms |
|                 │   ├─ TailAltsTimer from PileupMethods                     |     59.94 ms |            - |           - |       6 |   9.99 ms |   3.37 ms |  16.02 ms |
|                 │   ├─ TailCovTimer from PileupMethods                      |     10.64 ms |            - |           - |       6 |   1.77 ms | 321.07 µs |   6.53 ms |
|                 │   └─ TailEdgeTimer from PileupMethods                     |    326.72 µs |            - |           - |       6 |  54.45 µs |    8.7 µs | 269.95 µs |
|                 └─ AcumulatorAddTimer from PileupMethods                    |      98.7 µs |            - |           - |       6 |  16.45 µs |   9.66 µs |  44.17 µs |
| └─ BroadcastTimer from PileupMethods                                        |            - |    885.06 ms |   885.06 ms |       1 | 885.06 ms | 885.06 ms | 885.06 ms |
| └─ AdjustedEventsTimer from PileupMethods                                   |            - |      7.17 ms |     1.16 ms |       1 |   7.17 ms |   7.17 ms |   7.17 ms |
|     └─ map at ContigAggregateRDD.scala:46                                   |            - |      6.01 ms |           - |       1 |   6.01 ms |   6.01 ms |   6.01 ms |
|         └─ function call                                                    |    1.35 secs |            - |           - |       6 | 225.41 ms |   1.26 ms | 304.92 ms |
|             ├─ CalculateAltsTimer from ContigAggregate                      |    1.34 secs |            - |           - |       6 | 224.31 ms |  18.95 µs | 302.61 ms |
|             ├─ CalculateEventsTimer from ContigAggregate                    |      3.71 ms |            - |           - |       6 | 617.67 µs |  72.72 µs |   1.85 ms |
|             ├─ CalculateQualsTimer from ContigAggregate                     |    152.04 µs |            - |           - |       6 |  25.34 µs |  16.68 µs |   31.7 µs |
|             └─ ShrinkArrayTimer from PileupMethods                          |    126.24 µs |            - |           - |       6 |  21.04 µs |  12.86 µs |  46.44 µs |
| └─ EventsToPileupTimer from PileupMethods                                   |            - |     13.61 ms |     1.56 ms |       1 |  13.61 ms |  13.61 ms |  13.61 ms |
|     ├─ mapPartitions at ContigAggregateRDD.scala:51                         |            - |      6.26 ms |           - |       1 |   6.26 ms |   6.26 ms |   6.26 ms |
|     │   └─ function call                                                    |    181.09 ms |            - |           - |       6 |  30.18 ms |  23.18 ms |  53.94 ms |
|     └─ flatMap at ContigAggregateRDD.scala:100                              |            - |      5.79 ms |           - |       1 |   5.79 ms |   5.79 ms |   5.79 ms |
|         └─ function call                                                    |    206.36 µs |            - |           - |       6 |  34.39 µs |  22.77 µs |  71.74 µs |
+-----------------------------------------------------------------------------+--------------+--------------+-------------+---------+-----------+-----------+-----------+

Spark Operations
+----------+----------------------------------------------+---------------+----------------+--------------+----------+
| Sequence |                  Operation                   | Is New Stage? | Stage Duration | Driver Total | Stage ID |
+----------+----------------------------------------------+---------------+----------------+--------------+----------+
| 1        | first at AlignmentsRDD.scala:35              | false         |              - |    5.98 secs | -        |
| 2        | mapPartitions at AlignmentsRDD.scala:37      | false         |              - |     46.91 ms | -        |
| 3        | foreach at ContigAggregateRDD.scala:36       | false         |              - |     8.9 secs | -        |
| 4        | map at ContigAggregateRDD.scala:46           | false         |              - |      6.01 ms | -        |
| 5        | mapPartitions at ContigAggregateRDD.scala:51 | false         |              - |      6.26 ms | -        |
| 6        | flatMap at ContigAggregateRDD.scala:100      | false         |              - |      5.79 ms | -        |
+----------+----------------------------------------------+---------------+----------------+--------------+----------+