Skip to content

Commit

Permalink
Add process create and exit times (#47)
Browse files Browse the repository at this point in the history
* Working on adding exit codes, need to bump usersim version

* Update submodules

* Add tests and docs, try to fix annotation failure seen in CI

* Updating usersim to pick up SAL fix

* Add create and exit times, refactor test harness into .NET

* Add one more file to sln

* Trying to fix a project dependency issue seen in CI, and updating test docs a bit more

* Updating submodules, hopefully correctly this time

* Trying again with project dependency with the right path this time, to fix CI

* Adding MSBuild binlogs to better debug CI failures

* Adding more explicit dependencies to see if it fixes the race seen in CI

* Turn off native code analysis for managed code projects

* Continuing to try to fix tests running in CI, and address some code analysis warnings/messages

* Try to explicitly restore NuGets for the managed code in CI

* NuGet restore needs to happen after initialize_repo.ps1 so the submodules are updated first

* More cleanup of unused configurations, and ensuring NuGet restore is config-specific

* Fixing where the process monitor tests attempt to source ntosebpfext.sys from

* Need to fully qualify the path to ntosebpfext.sys

* Path to process_monitor.Tests.dll was wrong in the YAML
  • Loading branch information
Austin-Lamb authored May 12, 2024
1 parent 2d4b607 commit cbce18b
Show file tree
Hide file tree
Showing 39 changed files with 1,374 additions and 1,230 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/cicd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,8 @@ jobs:
uses: ./.github/workflows/reusable-test.yml
with:
name: process_monitor
pre_test: powershell .\Install-eBpfForWindows.ps1 0.16.0
test_command: powershell .\Test-ProcessMonitor.ps1
pre_test: powershell -file .\bin\process_monitor.Tests\win-x64\Install-eBpfForWindows.ps1 0.16.0 && powershell -file .\bin\process_monitor.Tests\win-x64\Setup-ProcessMonitorTests.ps1 -ArtifactsRoot .
test_command: dotnet test .\bin\process_monitor.Tests\win-x64\process_monitor.Tests.dll
build_artifact: Build-x64
environment: windows-2022
capture_etw: true
Expand Down
13 changes: 12 additions & 1 deletion .github/workflows/reusable-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,12 @@ jobs:
echo "C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
where clang.exe
- name: Install .NET 8 SDK
if: steps.skip_check.outputs.should_skip != 'true'
uses: actions/setup-dotnet@v3
with:
dotnet-version: 8.x

- name: Cache nuget packages
if: steps.skip_check.outputs.should_skip != 'true'
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9
Expand All @@ -154,10 +160,15 @@ jobs:
run: |
.\scripts\initialize_repo.ps1
- name: NuGet Restore
if: steps.skip_check.outputs.should_skip != 'true'
working-directory: ${{env.GITHUB_WORKSPACE}}
run: dotnet restore ${{env.SOLUTION_FILE_PATH}} /p:Configuration=${{env.BUILD_CONFIGURATION}} /p:Platform=${{env.BUILD_PLATFORM}}

- name: Build
if: steps.skip_check.outputs.should_skip != 'true'
working-directory: ${{env.GITHUB_WORKSPACE}}
run: msbuild /m /p:Configuration=${{env.BUILD_CONFIGURATION}} /p:Platform=${{env.BUILD_PLATFORM}} ${{env.BUILD_OPTIONS}} ${{env.SOLUTION_FILE_PATH}}
run: msbuild /m /p:Configuration=${{env.BUILD_CONFIGURATION}} /p:Platform=${{env.BUILD_PLATFORM}} /bl:${{env.BUILD_PLATFORM}}\${{env.BUILD_CONFIGURATION}}\build_logs\build.binlog ${{env.BUILD_OPTIONS}} ${{env.SOLUTION_FILE_PATH}}

- name: Zip Build Output
if: always() && (steps.skip_check.outputs.should_skip != 'true')
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,6 @@ artifacts/

# Visual Code.
**/.vscode/**/*

# MSBuild Binary Logs
*.binlog
27 changes: 22 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@ This will also allow us to better coordinate our efforts and minimize duplicated

To build locally, ensure your environment is set up:

1. Install Visual Studio 2022 with at least the "Desktop development with C++" workload
1. Install Visual Studio 2022 with at least the "Desktop development with C++" and ".NET desktop development" workloads
1. Install the .NET 8 SDK from https://dotnet.microsoft.com/en-us/download/dotnet/8.0
1. Install the Windows SDK 10.0.22621.0 with `winget install Microsoft.WindowsSDK.10.0.22621`
1. Install the Windows DDK 10.0.22621.0 with `winget install Microsoft.WindowsWDK.10.0.22621`

Expand All @@ -84,14 +85,30 @@ Run the unit tests by going to the binaries output folder (ex: `x64\Debug`) and

### E2E tests

The end-to-end tests use a tool called `process_monitor` to take data from the `ntosebpfext` extension and place it in a ring buffer that is visible from user-mode (this happens in `process_monitor.sys`). Then the `process_monitor.exe` user-mode process prints the events it sees to a file that the tests verify.
The end-to-end tests use a tool called `process_monitor` to take data from the `ntosebpfext` extension and place it in a ring buffer that is visible from user-mode (this happens in `process_monitor.sys`). Then the `process_monitor.exe` user-mode process prints the events it sees to the console. The `process_monitor.Tests` project contains MSTest tests that exercise the `process_monitor` code with an MSTest head instead of console output.

To run E2E tests you'll need to install eBPF for Windows and the ntosebpfext extension driver locally.

Do the following once:
1. Open a command prompt as admin
1. `cd <your binaries folder>` (ex: `<root of your clone>\x64\Debug`)
1. `powershell .\Install-eBpfForWindows.ps1 0.16.0`
1. `powershell .\Test-ProcessMonitor.ps1`
1. `cd <your local clone root>`
1. `cd x64\Debug\bin\process_monitor.Tests\win-x64`
1. `powershell -file .\Install-eBpfForWindows.ps1 0.16.0`
1. `powershell -file .\Setup-ProcessMonitorTests.ps1`

Then do this each time you want to re-run the tests:
1. `cd <your local clone root>`
1. `cd tests\process_monitor.Tests`
1. `dotnet test`

#### Running the E2E tests in Visual Studio

You can also run the tests in Visual Studio if it's running as admin. To do that:

1. Open the Test Explorer window (`Test -> Test Explorer`)
1. You may need to select the gear icon (which could be hidden behind a right arrow in the toolbar), and select the runsettings file (`RunSettings.runsettings` in the repo root). VS should auto-detect this though.
1. You may also want to select the gear icon and choose "run tests after build" to re-run each time you build.
1. Then just run the tests in Test Explorer by clicking the green "play" button. Or you can right-click a specific test and run/debug it.

### Debugging locally

Expand Down
43 changes: 18 additions & 25 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -4,35 +4,14 @@
SPDX-License-Identifier: MIT
-->
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" TreatAsLocalProperty="Platform">
<PropertyGroup Condition="'$(Analysis)'=='True'">
<PropertyGroup Condition="'$(Analysis)'=='True' And '$(MSBuildProjectExtension)'!='.csproj'">
<RunCodeAnalysis>true</RunCodeAnalysis>
<DisableAnalyzeExternal>true</DisableAnalyzeExternal>
<CodeAnalysisRuleSet>$(SolutionDir)Analyze.default.ruleset</CodeAnalysisRuleSet>
</PropertyGroup>
<PropertyGroup Condition="'$(AddressSanitizer)'=='True'">
<PropertyGroup Condition="'$(AddressSanitizer)'=='True' And '$(MSBuildProjectExtension)'!='.csproj'">
<EnableASAN>true</EnableASAN>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)'=='NativeOnlyDebug' Or '$(Configuration)'=='NativeOnlyRelease'">
<DisableJIT>true</DisableJIT>
<DisableInterpreter>true</DisableInterpreter>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)'=='Release' Or '$(Configuration)'=='NativeOnlyRelease'">
<FuzzerLibs>libsancov.lib;clang_rt.fuzzer_MD-x86_64.lib</FuzzerLibs>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='FuzzerDebug' Or '$(Configuration)'=='NativeOnlyDebug'">
<FuzzerLibs>libsancov.lib;clang_rt.fuzzer_MDd-x86_64.lib</FuzzerLibs>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Fuzzer)'=='Release|True'">
<EnableASAN>true</EnableASAN>
<AdditionalOptions>/fsanitize-coverage=inline-bool-flag /fsanitize-coverage=edge /fsanitize-coverage=trace-cmp /fsanitize-coverage=trace-div /ZH:SHA_256 %(AdditionalOptions)</AdditionalOptions>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Fuzzer)'=='Debug|True' Or '$(Configuration)'=='FuzzerDebug'">
<EnableASAN>true</EnableASAN>
<AdditionalOptions>/fsanitize-coverage=inline-bool-flag /fsanitize-coverage=edge /fsanitize-coverage=trace-cmp /fsanitize-coverage=trace-div /ZH:SHA_256 %(AdditionalOptions)</AdditionalOptions>
</PropertyGroup>
<PropertyGroup Condition="'$(Fuzzer)'!='True' And '$(Configuration)'!='FuzzerDebug'">
<SpectreMitigation>Spectre</SpectreMitigation>
</PropertyGroup>
<ItemDefinitionGroup>
<ClCompile>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
Expand Down Expand Up @@ -62,7 +41,7 @@
<PreprocessorDefinitions>CONFIG_BPF_INTERPRETER_DISABLED;%(ClCompile.PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)'=='Release' Or '$(Configuration)'=='NativeOnlyRelease'">
<ItemDefinitionGroup Condition="'$(Configuration)'=='Release'">
<ClCompile>
<RuntimeLibrary>MultiThreadedDLL</RuntimeLibrary>
<WholeProgramOptimization Condition="'$(EnableAsan)' != 'true'">true</WholeProgramOptimization>
Expand All @@ -72,11 +51,25 @@
<LinkTimeCodeGeneration Condition="'$(EnableAsan)' != 'true'">UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='FuzzerDebug' Or '$(Configuration)'=='NativeOnlyDebug'">
<ItemDefinitionGroup Condition="'$(Configuration)'=='Debug'">
<ClCompile>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<RuntimeLibrary>MultiThreadedDebugDLL</RuntimeLibrary>
</ClCompile>
</ItemDefinitionGroup>
<PropertyGroup Condition="'$(MSBuildProjectExtension)'=='.csproj'">
<RunSettingsFilePath>$(MSBuildThisFileDirectory)\RunSettings.runsettings</RunSettingsFilePath>
<TargetFramework>net8.0-windows</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<Platforms>x64</Platforms>
<PlatformTarget>x64</PlatformTarget>
<Platform>x64</Platform>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
<RuntimeIdentifiers>win-x64</RuntimeIdentifiers>
<UseArtifactsOutput>true</UseArtifactsOutput>
<ArtifactsPath>$(MSBuildThisFileDirectory)\$(Platform)\$(Configuration)</ArtifactsPath>
<ArtifactsPivots>$(RuntimeIdentifier)</ArtifactsPivots>
</PropertyGroup>
</Project>
20 changes: 20 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
<!--
Copyright (c) Microsoft Corporation
SPDX-License-Identifier: MIT
-->
<Project>
<PropertyGroup>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
</PropertyGroup>
<ItemGroup>
<!-- Try to keep these in alphabetical order -->
<PackageVersion Include="DotNet.ReproducibleBuilds" Version="1.1.1"/>
<PackageVersion Include="DotNet.ReproducibleBuilds.Isolated" Version="1.1.1"/>
<PackageVersion Include="eBPF-for-Windows" Version="0.16.0" />
<PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
<PackageVersion Include="Microsoft.Extensions.Logging.Console" Version="8.0.0" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
<PackageVersion Include="MSTest.TestAdapter" Version="3.2.2" />
<PackageVersion Include="MSTest.TestFramework" Version="3.2.2" />
</ItemGroup>
</Project>
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# NTOS eBPF Extensions

An eBPF for Windows extension that permits developers to leverage existing public hooks in the Windows kernel to gather data and influence policy of the OS.
An [eBPF for Windows](https://github.com/microsoft/ebpf-for-windows) extension that permits developers to leverage existing public
hooks in the Windows kernel to gather data and influence policy of the OS.

## Contributing

Expand Down
146 changes: 146 additions & 0 deletions RunSettings.runsettings
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
<?xml version="1.0" encoding="utf-8"?>
<!--
Copyright (c) Microsoft Corporation
SPDX-License-Identifier: MIT
-->
<!-- File name extension must be .runsettings -->
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<!-- Path relative to solution directory -->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<TargetPlatform>x64</TargetPlatform>
<TargetFrameworkVersion>.NETCoreApp, Version=8.0</TargetFrameworkVersion>

<!-- Timeout is in milliseconds, so this is 20 minutes, which should be vastly more than needed. -->
<TestSessionTimeout>1200000</TestSessionTimeout>
</RunConfiguration>

<DataCollectionRunSettings>
<DataCollectors>
<!-- The blame data collector will help attribute any tests that cause the test host process to crash -->
<DataCollector friendlyName="blame" enabled="True">
</DataCollector>

<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0"
assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CLRIEInstrumentationNetCore>True</CLRIEInstrumentationNetCore>
<CLRIEInstrumentationNetFramework>True</CLRIEInstrumentationNetFramework>
<UseManagedVanguard>True</UseManagedVanguard>
<CodeCoverage>
<!--
Additional paths to search for .pdb (symbol) files. Symbols must be found for modules to be instrumented.
If .pdb files are in the same folder as the .dll or .exe files, they are automatically found. Otherwise, specify them here.
Note that searching for symbols increases code coverage runtime. So keep this small and local.
-->
<!--
<SymbolSearchPaths>
<Path>C:\Users\User\Documents\Visual Studio 2013\Projects\ProjectX\bin\Debug</Path>
<Path>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
-->

<!--
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax). See http://msdn.microsoft.com/library/2k3te2cs.aspx.
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
-->

<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*process_monitor.*Tests\.dll$</ModulePath>
</Exclude>
</ModulePaths>

<!-- Match fully qualified names of functions: -->
<!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.) -->
<Functions>
<Exclude>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
<Function>^System.Text.RegularExpressions.Generated.*</Function>
</Exclude>
</Functions>

<!-- Match attributes on any code element: -->
<Attributes>
<Exclude>
<!-- Don�t forget "Attribute" at the end of the name -->
<Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
<Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
<Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
<Attribute>^NUnit.Framework.TestFixtureAttribute$</Attribute>
<Attribute>^Xunit.FactAttribute$</Attribute>
<Attribute>^Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute$</Attribute>
</Exclude>
</Attributes>

<!-- Match the path of the source files in which each method is defined: -->
<Sources>
<Exclude>
<Source>.*\\atlmfc\\.*</Source>
<Source>.*\\vctools\\.*</Source>
<Source>.*\\public\\sdk\\.*</Source>
<Source>.*\\microsoft sdks\\.*</Source>
<Source>.*\\vc\\include\\.*</Source>
</Exclude>
</Sources>

<!-- Match the company name property in the assembly: -->
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>

<!-- Match the public key token of a signed assembly: -->
<PublicKeyTokens>
<!-- Exclude Visual Studio extensions: -->
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>

<UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
<ForceVerifiableInstrumentation>False</ForceVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>False</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
<UseManagedInstrumentation>True</UseManagedInstrumentation>
<SkipNativeInstrumentation>True</SkipNativeInstrumentation>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>

<!-- Adapter Specific sections -->

<!-- MSTest adapter -->
<MSTest>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>true</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>true</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>True</DeploymentEnabled>
<InProcMode>false</InProcMode>
</MSTest>
</RunSettings>
4 changes: 3 additions & 1 deletion include/ebpf_ntos_hooks.h
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ typedef struct _process_md
uint64_t parent_process_id; ///< Parent process ID.
uint64_t creating_process_id; ///< Creating process ID.
uint64_t creating_thread_id; ///< Creating thread ID.
uint32_t process_exit_code; ///< Process exit status.
uint64_t creation_time; ///< Process creation time (as a FILETIME).
uint64_t exit_time; ///< Process exit time (as a FILETIME). Set only for PROCESS_OPERATION_DELETE.
uint32_t process_exit_code; ///< Process exit status. Set only for PROCESS_OPERATION_DELETE.
process_operation_t operation : 8; ///< Operation to do.
} process_md_t;

Expand Down
Loading

0 comments on commit cbce18b

Please sign in to comment.