Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ILM] Policy form should not throw away data #83077

Merged
merged 12 commits into from
Nov 20, 2020

Conversation

jloleysens
Copy link
Contributor

Summary

The ILM policy UI currently does not preserve unknown settings when serializing the form to the payload that will be sent to ES.

Notes

Given that the ILM policy UI does not cater for all configurations of ILM (e.g., the unfollow action) the following general strategy for ILM policy serialization (i.e., getting ready send back to ES) has been implemented:

  • Populate form fields with the policy as it is from ES (this is currently how the form works)
  • When serializing (i.e. getting ready to send back to ES), take the policy we can serialize from fields the UI knows about and merge it with a copy of the original policy -- this takes care of updating any fields we added
  • Clean up any known fields we removed

This has been implemented using immer and the merge function from lodash. This approach enables imperatively deleteing fields from the serialized object we know should not be included in the serialized output without modifying the original policy object.

How to review

  • Test loading up some ILM policies and editing them
  • Take a look at the "Show request" fly out for any irregularities
  • Look at the tests for what this fix is specifically addressed at
  • Please provide any feedback about the general approach taken

Checklist

@jloleysens jloleysens added Feature:ILM v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes v7.11.0 labels Nov 10, 2020
@jloleysens jloleysens requested a review from a team as a code owner November 10, 2020 16:39
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

Copy link
Contributor

@cjcenizal cjcenizal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't finished reviewing the code, but I will tomorrow. It's a bit slow going! I think I'm moving slowly because there are many complex conditions and the differences between the original and changed code are fairly subtle.

JL, I think your approach is sound: sanitize the output to ES during serialization by preserving unknown fields and removing known fields. I think I'd feel better about this if we had unit test coverage of serializer.ts that a) comprehensively defines the original expected behavior and b) allows us to verify that this behavior has been preserved and enhanced. We could even split it up into phase-level serialization functions for hot, warm, cold, and delete and add coverage at that level, to make it easier for readers to digest.

I know the client integration tests are passing, which reassures me that we haven't broken the behavior they test. After reading them in depth (which took a few minutes), I was able to associate their assertions with the different conditions in the serialization logic, but I need to go through them again to make sure I haven't missed anything. If we had a set of unit tests that made it easier to match inputs ("when warm phase is disabled") to outputs ("it's excluded from the serialized payload") I think my confidence level would go up that we hadn't broken anything, and maybe make review easier. How do you feel about this?

EDIT: So here's a possibly dumb idea about how we could test the behavior added in this PR. What if we had a test suite that dynamically generated tests that inserted an unknown action/setting in every possible valid location of a policy and verified that it wasn't stripped during serialization? Would this be a simple, focused, understandable way to verify that this code behaves as expected for this class of problem? Here's over-simplified pseudo-code to illustrate what I mean.

const policy = {
  name: 'policy',
  phases: {
    hot: {},
    warm: {},
    cold: {},
    delete: {},
  },
};

const validUnknownValueLocations = [
 '',
 'phases.hot',
 'phases.hot.actions',
 'phases.hot.actions.rollover',
 'phases.hot.actions.set_priority',
 'phases.warm',
 'phases.warm.actions',
 'phases.warm.actions.set_priority',
 // etc
];

validUnknownValueLocations.forEach(location => {
  it(`${location} preserves unknown property`, () => {
    const testPolicy = set(merge({}, policy), location, { unknown: 'unknown' });
    expect(serialize(testPolicy)).toEqual(testPolicy);
  });
});

// First copy over all non-allocate and migrate actions.
const actions: SerializedActionWithAllocation = { allocate, migrate, ...rest };
// First copy over all non-require|include|exclude and migrate actions.
const actions: SerializedActionWithAllocation = { ...rest };
Copy link
Contributor

@cjcenizal cjcenizal Nov 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused about what's being spread. The type indicates that newActions will only have allocate and migrate optional properties.

export interface SerializedActionWithAllocation {
  allocate?: AllocateAction;
  migrate?: MigrateAction;
}

So what's being copied over from the spread rest variable? Won't it always be undefined?

EDIT: 💡 Ah ha! Is this the mechanism we use to preserve unexpected properties provided by ES? If so, then I think a combination of comments and test coverage would have made that clearer to me as a first-time reader of the code. I think someone reading this code without the context provided by the PR description would have the same confusion noted above.

@@ -16,8 +20,16 @@ const serializeAllocateAction = (
originalActions: SerializedActionWithAllocation = {}
): SerializedActionWithAllocation => {
const { allocate, migrate, ...rest } = newActions;
// First copy over all non-allocate and migrate actions.
const actions: SerializedActionWithAllocation = { allocate, migrate, ...rest };
// First copy over all non-require|include|exclude and migrate actions.
Copy link
Contributor

@cjcenizal cjcenizal Nov 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of extracting this logic into a helper function, to encapsulate the operation? That would make it clearer to me what the operation is, and when reading the code I could examine it in isolation.

For example:

/**
 * Create an actions object with any actions that have been defined by ES but are unknown by the UI.
 */
const extractUnknownActions = (actions: SerializedActionWithAllocation) => {
  const { allocate, migrate, ...extractedUnkownActions } = actions;
  // First copy over all non-allocate and migrate actions.
  const unknownActions: SerializedActionWithAllocation = { ...extractedUnkownActions };

  // The UI only knows about include, exclude and require, so copy over all other values.
  if (allocate) {
    const { include, exclude, require, ...extractedUnknownAllocateSettings } = allocate;
    if (!isEmpty(extractedUnknownAllocateSettings)) {
      unknownActions.allocate = { ...extractedUnknownAllocateSettings };
    }
  }

  return unknownActions;
};

/* snip */

// line 22
const actions = extractUnknownActions(newActions);

Note that I also reverted the "First copy over all non-require|include|exclude and migrate actions" comment because the phrasing describes logic a few lines down, but not this line, which I found confusing.

}

if (_meta.hot.bestCompression && policy.phases.hot.actions?.forcemerge) {
policy.phases.hot.actions.forcemerge.index_codec = 'best_compression';
if (!updatedPolicy.phases.hot!.actions?.set_priority && hotPhaseActions.set_priority) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this could be simplified to:

if (!updatedPolicy.phases.hot!.actions?.set_priority) {
  delete hotPhaseActions.set_priority;
}

It might be personal preference, but the second condition makes me think I'm missing something when really... the second condition seems irrelevant?

- move serializer function around a little bit
- move serialize migrate and allocate function out of serializer
  file
Copy link
Contributor

@cjcenizal cjcenizal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes, JL! I tried introducing some bugs in the implementation and noticed that the tests didn't catch them. I think we need to increase the unit test coverage, WDYT?

[
'',
'phases.hot.actions',
'phases.hot.actions.rollover',
Copy link
Contributor

@cjcenizal cjcenizal Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dumb question: is this list intended to be comprehensive? I noticed phases.hot.actions.forcemerge, phases.warm.actions.allocate, phases.delete.actions.wait_for_snapshot, and others like that missing and it made me wonder.

Consider someone reading the serializeMigrateAndAllocateActions code and wanting to understand that behavior -- what can we do with this test to communicate that behavior to them?

Also, do we want to verify that unknown settings on phases.hot (e.g. phases.hot.unknown) and the other phase objects are preserved?

Copy link
Contributor Author

@jloleysens jloleysens Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I started writing this up and something didn't feel quite right because we would need to update this list in order for the check it is performing to remain thorough and reliable. I opted, instead, for creating a small bit of functionality that populates the entire policy with unknown values at all levels (except for one, but I left a comment about that). Let me know what you think!

const actions: SerializedActionWithAllocation = { ...otherActions };

// The UI only knows about include, exclude and require, so copy over all other values.
if (allocate) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a test, I pretended to be someone making changes to this code who didn't understand why we were copying "other actions" which aren't defined by our type system. I commented out lines 24 through 29 and replaced line 21 with:

// Hey, this is more efficient! I guess the original dev was just being extra safe?
const actions: SerializedActionWithAllocation = { ...newActions };

All of our tests still passed. I know this is a bit of a contrived example because it's likely someone would catch this change in CR or the dev would (hopefully) ask questions about the code, but if we're able to make this kind of significant implementation change without failing a test then I think there's a gap in the test coverage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this totally makes sense to me to add this coverage! I've added 6 additional test cases for how we might alter the form state, removing a field and how we expect this to come through when serialized.

delete warmPhase.actions.shrink;
}
} else {
delete draft.phases.warm;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I commented out this line, and all of our tests still passed.

}
} else {
delete hotPhaseActions.rollover;
delete hotPhaseActions.forcemerge;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to comment out this line without a test failure.

}

if (!updatedPolicy.phases.hot!.actions?.set_priority) {
delete hotPhaseActions.set_priority;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with this line.

if (_meta.cold.freezeEnabled) {
coldPhase.actions.freeze = {};
} else {
delete coldPhase.actions.freeze;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line too.

delete coldPhase.actions.set_priority;
}
} else {
delete draft.phases.cold;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line too.

@jloleysens jloleysens changed the title [ILM] Policy form does not throw away data [ILM] Policy form should not throw away data Nov 12, 2020
- removed the "forcemergeEnabled" meta field that was not being
  used
- added test cases for deleting of values from policies
@jloleysens
Copy link
Contributor Author

@cjcenizal Thanks for the review and sharing your thought process! I responded to your comments and made changes that I think address your points. Would you mind taking another look?

@jloleysens
Copy link
Contributor Author

@elasticmachine merge upstream

@jloleysens
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Contributor

@sebelga sebelga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not test locally the branch and I think @cjcenizal comments on commenting code and tests that keep on passing should be fixed as this is one of a king serializer! 😄

About the implementation (not blocker on this PR): I do find it easier to read this type of code with addition instead of subtraction. So I would read better this

const createSerializer = (originalPolicy) => (updatedPolicy) {
  const outputPolicy = {}; // start empty

  // Warm phase
  if (updatedPolicy.warm.enabled) {
    outputPolicy.warm = deepMerge({}, originalPolicy.warm, updatedPolicy.warm);
    
    // probably just a call to an external function
    serialzeWarmPhase(outputPolicy.warm, {} /* context object if needed */);
  }
  
  // Cold phase
  if (updatedPolicy.cold.enabled) {
    outputPolicy.cold = deepMerge({}, originalPolicy.cold, updatedPolicy.cold);
    serialzeColdPhase(outputPolicy.cold, {} /* context object if needed */);
  }

  return outputPolicy;
};

*/

import { setAutoFreeze } from 'immer';
import { cloneDeep } from 'lodash';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we use the lodash.clonedeep package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think not after #78156 was merged. We should be importing the entire lodash library now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could. Thanks, good to know 👍

Copy link
Contributor

@cjcenizal cjcenizal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work with these tests, JL! They're easy to read, and from reading them I can tell they're comprehensive. Thanks for adding them. I was still able to comment out some lines without any failing tests. I feel a bit like a chaos monkey... it's fun but I'd be surprised you're not annoyed by now! Sorry. 😅 🙈 Could you add tests to cover these?

In terms of preventing regressions, these tests were added as part of the change so we know we're protecting against regressions from this point onwards. How do we know this change doesn't introduce any regressions to the original behavior? From comparing the original code to the new code, the changes to createSerializer appear superficial, so I'm not concerned, but I wanted to ask how you verified this. For example, manual testing or through the existing client integration tests.

I also spotted some logic in createSerializer which doesn't appear to have corresponding logic in the original code. Are these bug fixes? Or am I overlooking something? If they're bug fixes, we need to add appropriate release notes to the PR description and labels to the PR.

I'm approving this to unblock you from merging once you've addressed my questions above. The changes and added code coverage look great over all! 👏

const populateWithUnknownEntries = (v: unknown) => {
if (isObject(v)) {
for (const key of Object.keys(v)) {
if (key === 'require' || key === 'include' || key === 'exclude') continue; // this will generate an invalid policy
Copy link
Contributor

@cjcenizal cjcenizal Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this helpful comment! This might be a bit easier to read (not a blocker though):

if (['require', 'include', 'exclude'].includes(key)) continue;

formInternal._meta.delete.enabled = false;

expect(serializer(formInternal)).toEqual({
...policy,
Copy link
Contributor

@cjcenizal cjcenizal Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line makes me read the definition of policy to understand what we're spreading and what will be retained. But it looks like the only thing we're retaining is the name property. I would find this easier to read if we didn't spread, and instead just hardcoded the name. WDYT? Just a suggestion, not a blocker.

    expect(serializer(formInternal)).toEqual({
      name: 'test',
      phases: {
        hot: policy.phases.hot, // We expect to see only the hot phase
      },
    });

) {
warmPhase.min_age = `${updatedPolicy.phases.warm!.min_age}${_meta.warm.minAgeUnit}`;
} else {
delete warmPhase.min_age;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to comment out this line, and the tests still passed.

}

if (!updatedPolicy.phases.warm!.actions?.shrink) {
delete warmPhase.actions.shrink;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line too.

Also, I don't see this corresponding logic in the original createSerializer. Is this newly added in this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's needed after we changed the strategy of merging with previously set values to ensure that we remove it. Also this work has not been released so we don't need to add fix release note :)

}

if (!updatedPolicy.phases.cold!.actions?.set_priority && coldPhase.actions.set_priority) {
delete coldPhase.actions.set_priority;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line too.

This is another line I don't see corresponding logic for in the original. Is it new?

@jloleysens
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
indexLifecycleManagement 110 112 +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
indexLifecycleManagement 211.0KB 212.4KB +1.4KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jloleysens jloleysens merged commit 6cce212 into elastic:master Nov 20, 2020
@jloleysens jloleysens deleted the ilm/fix-serializer branch November 20, 2020 09:18
jloleysens added a commit to jloleysens/kibana that referenced this pull request Nov 20, 2020
* fix ilm policy deserialization

* reorder expected jest object to match actual

* fix removal of wait for snapshot if it is not on the form

* add client integration test for policy serialization of unknown fields

* save on a few chars

* added unit test for deserializer and serializer

* Implement feedback

- move serializer function around a little bit
- move serialize migrate and allocate function out of serializer
  file

* Updated serialization unit test coverage

- removed the "forcemergeEnabled" meta field that was not being
  used
- added test cases for deleting of values from policies

* fixed minor issue in how serialization tests are being set up

Co-authored-by: Kibana Machine <[email protected]>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Nov 20, 2020
* master: (38 commits)
  [ML] Data frame analytics: Adds functionality to map view (elastic#83710)
  Add usage collection for savedObject tagging (elastic#83160)
  [SECURITY_SOLUTION] 145: Advanced Policy Tests (elastic#82898)
  [APM] Service overview transactions table (elastic#83429)
  [ML] Fix Single Metric Viewer not loading if job is metric with no partition (elastic#83880)
  do not export types from 3rd party modules as 'type' (elastic#83803)
  [Fleet] Allow to send SETTINGS action (elastic#83707)
  Fixes Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional_with_es_ssl/apps/triggers_actions_ui/details·ts - Actions and Triggers app Alert Details Alert Instances renders the active alert instances (elastic#83478)
  [Uptime]Reduce chart height on monitor detail page (elastic#83777)
  [APM] Prefer `APIReturnType` over `PromiseReturnType` (elastic#83843)
  [Observability] Fix telemetry for Observability Overview (elastic#83847)
  [Alerting] Adds generic UI for the definition of conditions for Action Groups (elastic#83278)
  ensure workload agg doesnt run until next interval when it fails (elastic#83632)
  [ILM] Policy form should not throw away data (elastic#83077)
  [Monitoring] Stop collecting Kibana Usage in bulkUploader (elastic#83546)
  [TSVB] fix wrong imports (elastic#83798)
  [APM] Correlations UI POC (elastic#82256)
  list all the refs in  tsconfig.json (elastic#83678)
  Bump jest (and related packages) to v26.6.3 (elastic#83724)
  Functional tests - stabilize reporting tests for cloud execution (elastic#83787)
  ...
jloleysens added a commit that referenced this pull request Nov 23, 2020
* fix ilm policy deserialization

* reorder expected jest object to match actual

* fix removal of wait for snapshot if it is not on the form

* add client integration test for policy serialization of unknown fields

* save on a few chars

* added unit test for deserializer and serializer

* Implement feedback

- move serializer function around a little bit
- move serialize migrate and allocate function out of serializer
  file

* Updated serialization unit test coverage

- removed the "forcemergeEnabled" meta field that was not being
  used
- added test cases for deleting of values from policies

* fixed minor issue in how serialization tests are being set up

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:ILM release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.11.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants