lazy load dsm only when needed #5305
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| const DataStreamsContext = require('./context') | ||
|
|
||
| class DataStreamsCheckpointer { | ||
| constructor (tracer) { | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| 'use strict' | ||
|
|
||
| const { | ||
| getAmqpMessageSize, | ||
| getHeadersSize, | ||
| getMessageSize, | ||
| getSizeOrZero | ||
| } = require('./size') | ||
|
|
||
| function lazyClass (classGetter, methods = [], staticMethods = []) { | ||
| let constructorArgs | ||
|
|
||
| const LazyClass = function (...args) { | ||
| constructorArgs = args | ||
| } | ||
|
|
||
| for (const method of methods) { | ||
| LazyClass.prototype[method] = function (...args) { | ||
| const ActiveClass = classGetter() | ||
| const instance = new ActiveClass(...constructorArgs) | ||
|
|
||
| Object.setPrototypeOf(this, instance) | ||
|
|
||
| return this[method](...args) | ||
| } | ||
| } | ||
|
|
||
| for (const method of staticMethods) { | ||
| LazyClass[method] = function (...args) { | ||
| const ActiveClass = classGetter() | ||
|
|
||
| for (const method of staticMethods) { | ||
| LazyClass[method] = ActiveClass[method] | ||
| } | ||
|
|
||
| return LazyClass[method](...args) | ||
| } | ||
| } | ||
|
|
||
| return LazyClass | ||
| } | ||
|
|
||
| const DsmPathwayCodec = lazyClass(() => require('./pathway').DsmPathwayCodec, [], [ | ||
| 'encode', | ||
| 'decode' | ||
| ]) | ||
|
|
||
| const DataStreamsCheckpointer = lazyClass(() => require('./checkpointer').DataStreamsCheckpointer, [ | ||
| 'setProduceCheckpoint', | ||
| 'setConsumeCheckpoint' | ||
| ]) | ||
|
|
||
| const DataStreamsManager = lazyClass(() => require('./manager').DataStreamsManager, [ | ||
| 'setCheckpoint', | ||
| 'decodeDataStreamsContext' | ||
| ]) | ||
|
|
||
| // TODO: Are all those methods actually public? | ||
| const DataStreamsProcessor = lazyClass(() => require('./processor').DataStreamsProcessor, [ | ||
| 'onInterval', | ||
| 'bucketFromTimestamp', | ||
| 'recordCheckpoint', | ||
| 'setCheckpoint', | ||
| 'recordOffset', | ||
| 'setOffset', | ||
| 'setUrl', | ||
| 'trySampleSchema', | ||
| 'canSampleSchema', | ||
| 'getSchema' | ||
| ]) | ||
|
|
||
| const SchemaBuilder = lazyClass(() => require('./schemas/schema_builder').SchemaBuilder, [ | ||
| 'build', | ||
| 'addProperty', | ||
| 'shouldExtractSchema' | ||
| ], [ | ||
| 'getCache', | ||
| 'getSchemaDefinition', | ||
| 'getSchema' | ||
| ]) | ||
|
|
||
| module.exports = { | ||
| DsmPathwayCodec, | ||
| DataStreamsCheckpointer, | ||
| DataStreamsManager, | ||
| DataStreamsProcessor, | ||
| SchemaBuilder, | ||
| getAmqpMessageSize, | ||
| getHeadersSize, | ||
| getMessageSize, | ||
| getSizeOrZero | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| 'use strict' | ||
|
|
||
| const { DsmPathwayCodec } = require('./pathway') | ||
| const DataStreamsContext = require('./context') | ||
|
|
||
| class DataStreamsManager { | ||
| constructor (processor) { | ||
| this._dataStreamsProcessor = processor | ||
| } | ||
|
|
||
| setCheckpoint (edgeTags, span, payloadSize = 0) { | ||
| const ctx = this._dataStreamsProcessor.setCheckpoint( | ||
| edgeTags, span, DataStreamsContext.getDataStreamsContext(), payloadSize | ||
| ) | ||
| DataStreamsContext.setDataStreamsContext(ctx) | ||
| return ctx | ||
| } | ||
|
|
||
| decodeDataStreamsContext (carrier) { | ||
| const ctx = DsmPathwayCodec.decode(carrier) | ||
| // we erase the previous context everytime we decode a new one | ||
| DataStreamsContext.setDataStreamsContext(ctx) | ||
| return ctx | ||
| } | ||
| } | ||
|
|
||
| module.exports = { DataStreamsManager } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| 'use strict' | ||
|
There was a problem hiding this comment. The helpers in this file were copied from |
||
|
|
||
| const { types } = require('util') | ||
|
|
||
| function getSizeOrZero (obj) { | ||
| if (typeof obj === 'string') { | ||
| return Buffer.from(obj, 'utf-8').length | ||
| } | ||
| if (types.isArrayBuffer(obj)) { | ||
| return obj.byteLength | ||
| } | ||
| if (Buffer.isBuffer(obj)) { | ||
| return obj.length | ||
| } | ||
| if (Array.isArray(obj) && obj.length > 0) { | ||
| if (typeof obj[0] === 'number') return Buffer.from(obj).length | ||
| let payloadSize = 0 | ||
| obj.forEach(item => { | ||
| payloadSize += getSizeOrZero(item) | ||
| }) | ||
| return payloadSize | ||
| } | ||
| if (obj !== null && typeof obj === 'object') { | ||
| try { | ||
| return getHeadersSize(obj) | ||
| } catch { | ||
| // pass | ||
| } | ||
| } | ||
| return 0 | ||
| } | ||
|
|
||
| function getHeadersSize (headers) { | ||
| if (headers === undefined) return 0 | ||
| return Object.entries(headers).reduce((prev, [key, val]) => getSizeOrZero(key) + getSizeOrZero(val) + prev, 0) | ||
| } | ||
|
|
||
| function getMessageSize (message) { | ||
| const { key, value, headers } = message | ||
| return getSizeOrZero(key) + getSizeOrZero(value) + getHeadersSize(headers) | ||
| } | ||
|
|
||
| function getAmqpMessageSize (message) { | ||
| const { headers, content } = message | ||
| return getSizeOrZero(content) + getHeadersSize(headers) | ||
| } | ||
|
|
||
| module.exports = { | ||
| getMessageSize, | ||
| getHeadersSize, | ||
| getSizeOrZero, | ||
| getAmqpMessageSize | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a comment that explains how this replaces all prototype replaced methods in one go.
Please add a separate TODO (including a time frame for a lintrule similar to how unicorn does it https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/expiring-todo-comments.md) that this is going to be replaced with a system that encapsulates modules and have dedicates plugins for that module or feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually writing the comments made me realize the code can be simplified a bit so I refactored it. I also added the comments and TODO as requested, but I didn't add a time frame as this is not really something we do today, and I felt like this would need to be properly discussed and the linter capability added first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any other concerns with the PR?