Skip to content

Commit 0168747

Browse files
nth-commitnth-commit
committed
🌍 #376 Refactor - combine the sync and async check state machines
This was a four month long battle. I picked it up many times throughout, each time being tempted by `git branch -D terra/async-refactor`. Previously, one of the core pieces of infrastructure in GC was copy+pasted between Check/CheckAsync, it was a lot of lines of code to keep in sync. Even though CheckAsync is the primary usecase (it's the entrypoint used by GalaxyCheck.Xunit), I was not keen to drop the synchronous overload, as it would be a giant pain to litter the codebase with async/await. This also means we can mostly forget about testing the async version of the code, as it barely deviates from the sync version at all, only at the entrypoint in `CheckAsync`.
1 parent 8dfd600 commit 0168747

File tree

85 files changed

+1597
-2184
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

85 files changed

+1597
-2184
lines changed

‎GalaxyCheck.sln.DotSettings

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
2+
<s:Boolean x:Key="/Default/UserDictionary/Words/=Nullary/@EntryIndexedValue">True</s:Boolean></wpf:ResourceDictionary>

‎docs/deprecated-features.md

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Deprecated Features
2+
3+
This doc is intended to capture (experimental) features that were discarded, and why.
4+
5+
## Preconditions
6+
7+
This was a convenience feature that allowed you to specify a precondition inside the test body, which simplified the way you would write some properties. Consider the following:
8+
9+
```
10+
private static IGen<(int x, int y)> TwoDistinctIntegers =>
11+
from x in Gen.Int32()
12+
from y in Gen.Int32()
13+
where x != y
14+
select (x, y);
15+
16+
[Property]
17+
public void DifferentIntegersHashDifferently([MemberGen(nameof(TwoDistinctIntegers))] (int x, int y) tupleOfInts)
18+
{
19+
Assert.NotEqual(Hash(tupleOfInts.x), Hash(tupleOfInts.y));
20+
}
21+
```
22+
23+
This could be expressed with preconditions as:
24+
25+
```
26+
[Property]
27+
public void DifferentIntegersHashDifferently(int x, int y)
28+
{
29+
Property.Precondition(x != y);
30+
31+
Assert.NotEqual(Hash(x), Hash(y));
32+
}
33+
```
34+
35+
This was deprecated because it added significant complexity to GalaxyCheck. It is useful, and may be added in the future in a more considered way.
36+
37+
The main problems were:
38+
39+
1. Firstly, filtering generators is just hard. All generators have an input size, which determines the range of the values they produce. A size of 0 will always produce 0 values (empty lists, strings, or literally a `0` for integers). We have some rough heuristics for "resizing" if the generator fails to produce a value - for example the above property will never be satisfied if the size is 0 (both x and y will = 0).
40+
2. Implementing pre-conditions into GalaxyCheck as it stands required separate, disconnected, resizing heuristics in the generators (such as in the first example above), and in the test runner (the `Check` function), which decorates generators and provides the state machine for iterating through test-cases. A future solution would consider a "holistic" filtering approach, where the behaviour is injected into both models, and we have a single state that holds filter metrics.
41+
3. Samples. An core value of the library is that it should be observable, so that an appropriate level of trust can be built between a user and the library. One of the main features that enables this is the ability to easily sample what values _would have_ been passed to your test, and you can do this by swapping a `Sample` attribute in for a `Property` attribute. The interaction between sampling and preconditions needs special consideration (in implementation and desired behaviour), as a value is passed to your test, then it is discard. Samples of the two properties above would produce different results, if not especially considered by the library.

‎src/GalaxyCheck.Tests.V3/DomainGen.cs

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
using NebulaCheck;
2+
3+
namespace GalaxyCheck_Tests_V3;
4+
5+
public static class DomainGen
6+
{
7+
public static class TestFunctions
8+
{
9+
public static IGen<Func<bool>> Nullary() => Gen.Function(Gen.Boolean());
10+
11+
public static IGen<Func<T, bool>> Unary<T>() => Gen.Function<T, bool>(Gen.Boolean());
12+
}
13+
14+
public static class Gens
15+
{
16+
public static IGen<GalaxyCheck.IGen<object>> Error() => Gen
17+
.String()
18+
.WithLengthBetween(0, 10)
19+
.Select(GalaxyCheck.Gen.Advanced.Error<object>);
20+
}
21+
22+
public static class Properties
23+
{
24+
public static IGen<PropertyProxy> FromFunction(Func<bool> testFunction) =>
25+
from seed in Seed()
26+
from size in Size()
27+
select new PropertyProxy(seed, size, _ => testFunction());
28+
29+
public static IGen<PropertyProxy> FromFunction(Func<object, bool> testFunction) =>
30+
from seed in Seed()
31+
from size in Size()
32+
select new PropertyProxy(seed, size, testFunction);
33+
34+
public static IGen<PropertyProxy> Any() =>
35+
from seed in Seed()
36+
from size in Size()
37+
from testFunction in TestFunctions.Unary<object>()
38+
select new PropertyProxy(seed, size, testFunction);
39+
40+
public static IGen<GalaxyCheck.Property> Nullary() =>
41+
from testFunction in TestFunctions.Nullary()
42+
select GalaxyCheck.Property.Nullary(testFunction);
43+
}
44+
45+
public static IGen<int> Size() => Gen.Int32().Between(GalaxyCheck.Gens.Parameters.Size.Zero.Value, GalaxyCheck.Gens.Parameters.Size.Max.Value);
46+
47+
public static IGen<int> Seed() => Gen.Int32();
48+
49+
public static IGen<int?> SeedWaypoint() => Seed().OrNullStruct();
50+
51+
public static IGen<GalaxyCheck.Property> ToProperty(this IGen<GalaxyCheck.IGen<object>> metaGen) =>
52+
from gen in metaGen
53+
from testFunction in TestFunctions.Unary<object>()
54+
select new GalaxyCheck.Property((GalaxyCheck.IGen<GalaxyCheck.Property.Test<object>>)GalaxyCheck.Property.ForAll<object>(gen, testFunction));
55+
}

‎src/GalaxyCheck.Tests.V3/Features/Generators/Operators/AboutWhereOperator.cs

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
using GalaxyCheck;
2+
3+
namespace GalaxyCheck_Tests_V3.Features.Generators.Operators;
4+
5+
public class AboutWhereOperator
6+
{
7+
[NebulaCheck.Property]
8+
public NebulaCheck.Property ItProducesValuesThatPassThePredicate()
9+
{
10+
return NebulaCheck.Property.ForAll(DomainGen.Seed(), DomainGen.Size(), Test);
11+
12+
static void Test(int seed, int size)
13+
{
14+
// Arrange
15+
var testFunction = DummyTestFunctions.Int32.NonZero();
16+
var baseGen = Gen.Int32();
17+
var filteredGen = baseGen.Where(testFunction);
18+
19+
// Act
20+
var sample = filteredGen.Sample(args => args with { Seed = seed, Size = size });
21+
22+
// Assert
23+
sample.Should().OnlyContain(x => testFunction(x));
24+
}
25+
}
26+
27+
[NebulaCheck.Property]
28+
public NebulaCheck.Property ItCanAdaptToPredicatesThatOnlyPassForLargerSizes()
29+
{
30+
return NebulaCheck.Property.ForAll(DomainGen.Seed(), DomainGen.Size(), Test);
31+
32+
static void Test(int seed, int size)
33+
{
34+
// Arrange
35+
var testFunction = DummyTestFunctions.Size.Top50thPercentile();
36+
var baseGen = DummyGens.TheInputSize();
37+
var filteredGen = baseGen.Where(testFunction);
38+
39+
// Act
40+
var act = () => filteredGen.Sample(args => args with { Seed = seed, Size = size });
41+
42+
// Assert
43+
act.Should().NotThrow<Exceptions.GenExhaustionException>();
44+
}
45+
}
46+
}

‎src/GalaxyCheck.Tests.V3/Features/Replaying/AboutReplayEncoding.cs

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
using System.Runtime.InteropServices;
2+
using GalaxyCheck.Gens.Parameters;
3+
using GalaxyCheck.Gens.Parameters.Internal;
4+
using GalaxyCheck.Runners.Replaying;
5+
6+
namespace GalaxyCheck_Tests_V3.Features.Replaying;
7+
8+
public class AboutReplayEncoding
9+
{
10+
public static TheoryData<int, int, int?, IEnumerable<int>, string> Data =>
11+
new()
12+
{
13+
{
14+
0, 0, null, new[] { 0 },
15+
"H4sIAAAAAAAACjPQM9AzAACGS1suBQAAAA=="
16+
},
17+
{
18+
0, 0, null, new[] { 0, 0, 0, 0, 0, 0 },
19+
"H4sIAAAAAAAACjPQM9AzsIJDALKWW5IPAAAA"
20+
},
21+
{
22+
0, 0, null, Enumerable.Range(0, 10).Select(_ => 0),
23+
"H4sIAAAAAAAACjPQM9AzsMKAANukRbAXAAAA"
24+
},
25+
{
26+
0, 0, null, Enumerable.Range(0, 100).Select(_ => 0),
27+
"H4sIAAAAAAAACjPQM9AzsBoWEAC35Wk5ywAAAA=="
28+
},
29+
{
30+
int.MaxValue, 100, int.MaxValue, Enumerable.Range(0, 100),
31+
"H4sIAAAAAAAACj2Qxw0EQQzDOlo4yIn9F3bzuq8ABSpco83WfG72GU6QiKIZlsOf6HjgiQsvvPHBFz/CiOcJIgkRRTQxxBJHGunki0xSZJFNDrnkIUOOAr1GoUKNBi06yiingkrqDSqqqaGWOtpop4NOWvTb2/TQSx9jjDPBJCOmmIczzDLHGutssMmKLbbZR7vsccY5F1xy4oprbrh3xn3xP+wHz39ZWzsBAAA="
32+
},
33+
};
34+
35+
[Theory]
36+
[MemberData(nameof(Data))]
37+
public void Examples(int seed, int size, int? seedWaypoint, IEnumerable<int> exampleSpacePath, string expectedEncoding)
38+
{
39+
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows) is false)
40+
{
41+
// TODO: Cross-platform support for this behaviour - I think this might be fixed in .NET 7
42+
return;
43+
}
44+
45+
var replay = CreateReplay(seed, size, seedWaypoint, exampleSpacePath);
46+
47+
var encoded = ReplayEncoding.Encode(replay);
48+
49+
encoded.Should().Be(expectedEncoding);
50+
}
51+
52+
[NebulaCheck.Property]
53+
public NebulaCheck.Property EncodeDecodeRoundtrip()
54+
{
55+
return NebulaCheck.Property.ForAll(DomainGen.Seed(), DomainGen.Size(), DomainGen.SeedWaypoint(), FeatureGen.ShrinkPath(), Test);
56+
57+
static void Test(int seed, int size, int? seedWaypoint, IEnumerable<int> exampleSpacePath)
58+
{
59+
var replay0 = CreateReplay(seed, size, seedWaypoint, exampleSpacePath);
60+
var replay1 = ReplayEncoding.Decode(ReplayEncoding.Encode(replay0));
61+
62+
replay1.Should().BeEquivalentTo(replay0);
63+
}
64+
}
65+
66+
private static Replay CreateReplay(int seed, int size, int? seedWaypoint, IEnumerable<int> exampleSpacePath) =>
67+
new(new GenParameters(Rng.Create(seed), new Size(size), seedWaypoint == null ? null : Rng.Create(seedWaypoint.Value)), exampleSpacePath);
68+
}

‎src/GalaxyCheck.Tests.V3/Features/Replaying/AboutReplayingProperties.cs

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
using GalaxyCheck;
2+
using GalaxyCheck.Runners.Replaying;
3+
4+
namespace GalaxyCheck_Tests_V3.Features.Replaying;
5+
6+
public class AboutReplaying
7+
{
8+
[Fact]
9+
public void ItOnlyCallsTheTestFunctionOnce()
10+
{
11+
// Arrange
12+
var spy = new Spy<int>();
13+
var property = DummyProperties.EventuallyFalsified(spy);
14+
var counterexample = property.EnsureFalsified();
15+
spy.Reset();
16+
17+
// Act
18+
property.Check(args => args with { Replay = counterexample.Replay });
19+
20+
// Assert
21+
spy.Values
22+
.Should().ContainSingle()
23+
.Which.Should().Be(counterexample.Value);
24+
}
25+
26+
[NebulaCheck.Property]
27+
public NebulaCheck.Property IfReplayIsAnInvalidFormat_ItThrows()
28+
{
29+
return NebulaCheck.Property.ForAll(FeatureGen.NotBase64(), DomainGen.Properties.Any(), Test);
30+
31+
static void Test(string replay, PropertyProxy property)
32+
{
33+
// Act
34+
Action test = () => property.Check(args => args with { Replay = replay });
35+
36+
// Assert
37+
test.Should()
38+
.Throw<Exceptions.GenErrorException>()
39+
.WithMessage("Error decoding replay string:*");
40+
}
41+
}
42+
43+
[NebulaCheck.Property]
44+
public NebulaCheck.Property IfReplayEncodesAnInvalidPath_ItThrows()
45+
{
46+
return NebulaCheck.Property.ForAll(FeatureGen.Replay(allowEmptyPath: false), DomainGen.Properties.Nullary(), Test);
47+
48+
static void Test(Replay replay, Property property)
49+
{
50+
// Act
51+
Action test = () => property.Check(args => args with { Replay = ReplayEncoding.Encode(replay) });
52+
53+
// Assert
54+
test.Should()
55+
.Throw<Exceptions.GenErrorException>()
56+
.WithMessage("Error replaying last example, given replay string was no longer valid.*");
57+
}
58+
}
59+
60+
[NebulaCheck.Property]
61+
public NebulaCheck.Property IfReplayEncodesAnError_ItThrows()
62+
{
63+
return NebulaCheck.Property.ForAll(FeatureGen.Replay(), DomainGen.Gens.Error().ToProperty(), Test);
64+
65+
static void Test(Replay replay, Property property)
66+
{
67+
// Act
68+
Action test = () => property.Check(args => args with { Replay = ReplayEncoding.Encode(replay) });
69+
70+
// Assert
71+
test.Should()
72+
.Throw<Exceptions.GenErrorException>()
73+
.WithMessage("Error during generation:*");
74+
}
75+
}
76+
}

‎src/GalaxyCheck.Tests.V3/Features/Replaying/FeatureGen.cs

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
using NebulaCheck;
2+
using NebulaCheck.Gens;
3+
4+
namespace GalaxyCheck_Tests_V3.Features.Replaying;
5+
6+
internal static class FeatureGen
7+
{
8+
public static IGen<string> NotBase64() => Gen.Constant("0");
9+
10+
public static IGen<GalaxyCheck.Runners.Replaying.Replay> Replay(bool allowEmptyPath = true) =>
11+
from seed in DomainGen.Seed()
12+
from size in DomainGen.Size()
13+
from path in ShrinkPath(allowEmptyPath)
14+
select new GalaxyCheck.Runners.Replaying.Replay(GalaxyCheck.Gens.Parameters.GenParameters.Parse(seed, size), path);
15+
16+
public static IGen<IReadOnlyList<int>> ShrinkPath(bool allowEmpty = true) =>
17+
allowEmpty ? Gen.Int32().ListOf() : Gen.Int32().ListOf().WithCountGreaterThanEqual(1);
18+
}

‎src/GalaxyCheck.Tests.V3/GalaxyCheck.Tests.V3.csproj

+6-6
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@
1111
</PropertyGroup>
1212

1313
<ItemGroup>
14-
<PackageReference Include="FluentAssertions" Version="6.11.0"/>
15-
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2"/>
16-
<PackageReference Include="NebulaCheck.Xunit" Version="0.0.0-4021874656"/>
17-
<PackageReference Include="xunit" Version="2.4.2"/>
14+
<PackageReference Include="FluentAssertions" Version="6.11.0" />
15+
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
16+
<PackageReference Include="NebulaCheck.Xunit" Version="0.0.0-4021874656" />
17+
<PackageReference Include="xunit" Version="2.4.2" />
1818
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
1919
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
2020
<PrivateAssets>all</PrivateAssets>
@@ -26,8 +26,8 @@
2626
</ItemGroup>
2727

2828
<ItemGroup>
29-
<ProjectReference Include="..\GalaxyCheck.Xunit\GalaxyCheck.Xunit.csproj"/>
30-
<ProjectReference Include="..\GalaxyCheck\GalaxyCheck.csproj"/>
29+
<ProjectReference Include="..\GalaxyCheck.Xunit\GalaxyCheck.Xunit.csproj" />
30+
<ProjectReference Include="..\GalaxyCheck\GalaxyCheck.csproj" />
3131
</ItemGroup>
3232

3333
</Project>

‎src/GalaxyCheck.Tests.V3/NebulaCheckConfiguration.cs

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
using NebulaCheck.Configuration;
2+
3+
namespace GalaxyCheck_Tests_V3;
4+
5+
public class NebulaCheckConfiguration : IConfigureGlobal
6+
{
7+
public void Configure(IGlobalConfiguration instance)
8+
{
9+
instance.Properties.DefaultIterations = 10;
10+
}
11+
}

0 commit comments

Comments
 (0)