Feature request - return TestCases property as part of TestResults

[nvarscar]

1. General summary of the issue

It would be really nice if there was an option to see which objects have been tested within the It block. There is -TestCases parameter that can dynamically change the It block name, but there is no way to access dictionary items that were used during the execution, except for parsing the test name string.

3. Expected Behavior

Each TestResult in the pester output (-Passthru) would have one more property, say, TestCase, which would include the dictionary item used for the If block, or just a custom PSObject. Such information could have been used further down the execution scenario:

• knowing server/service names, restart a frozen website
• choose a proper fix scenario for an incorrect configuration item, having all current item details

4.Current Behavior

The only way to obtain such information right now is to parse a test name, which unfortunately is far from ideal. Having to use a custom parsing string proves to be more inconvenient than it could've been if the original values were passed along as an object.

5. Possible Solution

Add corresponding TestCase object into the output of each TestResult
(Optionally, for user convenience) Allow TestCases a wider range of object types, preferably, [object[]]

6. Context

I want to be able to, after discovering that a certain item of our infrastructure is not configured properly (or not responding), pass the results of the test to a function that will be able to fix such issue, based on test type and affected object details returned with the test results.

[nohwnd]

@nvarscar Could you clarify this please? "Optionally, for user convenience) Allow TestCases a wider range of object types, preferably, [object[]]"

How should that work? Currently we use hashtable in TestCases because we need the keys to be able to expand the names into the test descriptions.

[nvarscar]

@nohwnd the only way I see it working is adding an alternative way of parsing the properties using psobject parameter, which opens access to all of the object's parameters:

$object = Test-NetConnection -ComputerName localhost -Port 1234
$object.psobject.properties | select name, value

Not elegant and more of a "one size fits all" kind of approach, also does not work with hashtables so well, which is why [IDictionary] objects should probably be still parsed as they are right now.

[nohwnd]

@nvarscar I am still lost. The example did not help me. :/

What I understood is that you want to add the current test case to the result object. So having test cases like this:

@(
    @{ Value = 'a';  Expected = 'A' },
    @{ Value = 'b';  Expected = 'B' }
)

The test result object would contain property TestCase with @{ Value = 'a'; Expected = 'A' } in it.

---

Then I also understood that you want to pass array of objects to the TestCases parameter, but I don't know, if you know that you can already pass any object to the test cases, you just need to wrap it in the hashtable and give it some key. So with the above change implemented you'd be able to pass any data to have them in the result object.

@(
    @{ Value = 'a';  Expected = 'A' , SomeOtherData = <whatever object you want>}
)

So my question is, I am missing your point? If I do then please show me example that includes the usage of -TestCases and spell out the details :)

[nvarscar]

@nohwnd you understood it perfectly fine! 😄
now, here's some real-life example of passing an object[] to the TestCases:

Describe 'test connections' {
  $a = test-connection -ComputerName localhost,localhost -Count 1
  It 'should response within 2ms' -TestCases $a {
    Param (
      $inputObject
    )
    $inputObject.responseTime | Should -BeLessThan 3
  }
}

That would probably require defining an input object parameter instead of splatting (with pre-defined name), because now that I think about it, I don't see how splatting would work here. So, disregard my previous comment about using .psobject.properties - splatting all possible properties of the object would require users to include all of them in the test, which is not completely desirable. It still could work with -TestCases ($object | Select Property1, Property2)) though, but it would be less obvious how to use the parameter.

[nohwnd]

@nvarscar is there a parameter missing in your example? Because $a is not used for anything. Did you mean -TestCases $a?

[nvarscar]

@nohwnd correct, thanks for noticing! edited the code. I'm still not completely awake, apparently.

[nohwnd]

Currently you can do this:

Describe 'test connections' {
  $a = test-connection -ComputerName google.com, localhost -Count 1 |
    % { @{ Address = $_.Address; InputObject = $_ }} 

  It -TestCases $a '<address> should respond within 2ms' {
    Param (
      $InputObject
    )
    $InputObject.responseTime | Should -BeLessThan 3
  }
}

To me it seems reasonably simple and descriptive. But making TestCases object[] and expanding first-level properties if we get object, or keys if we get IDictionary, should also by quite simple and non-confusing. What I am not sure about is doing multi level properties, it seems simple, but I am sure someone will come up with some horrible edge case as soon as it's out of beta, such as the first property being array, or the property is named [abcd.ggggg]::prop#```1 😄 So I would avoid that, and the caller will simply need to format the object correctly himself - most likely using hashtable as in the example.

Also there is the idea of executing any code in the test description and the idea of special formatters like <inputValue:count> or <inputValue:type> in nohwnd/Assert#14

[nvarscar]

I guess, that works, as long as the proposed feature gets implemented 🏭

[nvarscar]

I just verified the example you had and it seems that there is already something like this implemented:

$p = Invoke-Pester <file above> -PassThru
$p.TestResult.Parameters.InputObject #<= this is the object I was looking for

Somehow, Address did not make it though, but that's completely fine - I will do more testing and close this issue if it works as expected.

[nvarscar]

So yes, I can confirm that all the parameters mentioned in the Param () section are becoming a part of the .Parameters property, which makes sense.. Though I expected that the whole object would make it to the testResults, not just the part that is mentioned in the It scriptblock (and it is the reason why I thought it's not being passed at all).
I will close the issue, as this is exactly what I needed.

[nohwnd]

Keeping this open so we can implement the first part (at least).

[michalporeba]

passing object[] and expanding the first level properties would work very well in my use-cases of environmental checks. Frequently I'm getting some sort of object, which then I have to translate to a hashtable before I can execute them as test cases. It's awkward and, for example in dbachecks project instead of doing these variables are set outside of It and read inside it (typically in a loop) instead of use cases, which looks ugly too.

Allowing for an easy way to pass complex objects to It as test cases can help with the use of independent tests as described in #721.

Having already support for object[] and IDictionary, it should be trivial to add support for string[] from the example in #832

[nohwnd]

Reviewing this, my concern is that objects potentially have a ton of properties and all of them will get expanded in the upper scope (in v5) so you don't have to add param block. This might be slow, and shadow your variables, so a way to skip some variables would be needed. I could for example add Get-TestCase cmdlet that you give an object to, and properties to include/skip), and then detect if I am getting an object with known pstypename (like "PesterTestCasesDescriptor") or just a normal object.

There is other stuff like should it expand fields as well? As you can see it quickly gets complicated and so I would rather stick with hashtables for now, and let you transform your objects as you please.

[nohwnd]

$result = Invoke-Pester -ScriptBlock {
    Describe 'Passing test cases' {

        It 't1 <value>' -TestCases @(
            @{
                Value = "This one I use"
                Extra = @{
                    This = "I save"
                    For = "Later"
                }
            }
        ) {

            $Value | Should -Be "This one I use"
        }
    }
} -PassThru

$result.Blocks[0].Tests[0].Data

## Output
# Found 1 tests
# Processing discovery result objects, to set root, parents, filters etc.
# Test discovery finished.
# 
# Describing Passing test cases
#     [+] t1 This one I use 22ms
# Tests completed in 182ms
# Tests Passed: 1, Failed: 0, Skipped: 0, Pending: 0, Inconclusive: 0 
# 
# Name                           Value
# ----                           -----
# Extra                          {This, For}
# Value                          This one I use

Works in v5, the whole hashtable the testcase got is persisted in the result object, no matter which values are used in the test.

I won't add the implicit expansion of properties on object, because in combination with automatic parameter blocks it would lead to the problems described above, and it is easy to make your own hashtable from an object.