Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Agentic memory #5227

Open
wants to merge 105 commits into
base: main
Choose a base branch
from
Open

Agentic memory #5227

wants to merge 105 commits into from

Conversation

rickyloynd-microsoft
Copy link
Contributor

@rickyloynd-microsoft rickyloynd-microsoft commented Jan 28, 2025 •
edited
Loading

Adds a baseline implementation of agentic memory to the autogen-ext package.

For technical details, see the Agentic Memory README.

To see it in action, view the page logs generated while running the code samples.

Items we've discussed that may be handled in post-PR issues:

  • Logging to *.jsonl files.
  • Object serialization and deserialization.
  • Making AgenticMemoryBank use a chromadb-specific implementation of autogen-core/memory.

@rickyloynd-microsoft
initial checkin
442a9d8
@rickyloynd-microsoft
support for extensive evaluations
f8584cd
@rickyloynd-microsoft
Enhance retrieval with task generalization and insight validation
607e7ff
@rickyloynd-microsoft
Support TRAPI client.
b045636 
Make memory optional.
Filter out insights with negative scores.
@rickyloynd-microsoft
Restoring earlier results, and general cleanup.
63b28d7
@rickyloynd-microsoft
Merge branch 'refs/heads/main' into agentic_memory
b921d83
@rickyloynd-microsoft
Modify imports after merge from main.
9dfb074
@rickyloynd-microsoft
Log model and token counts.
93a5ca4
@rickyloynd-microsoft
Only instantiate the client once.
2cb9344
@rickyloynd-microsoft
Fix bug that was duplicating insights across trials.
878f458
@rickyloynd-microsoft
Add the Grader class.
21562f1 
Refactor memory paths.
Enrich page logging.
@rickyloynd-microsoft
Adjustments for comparison tests.
3a40b30
@rickyloynd-microsoft
Test generalization over multiple tasks.
8622c5e
@rickyloynd-microsoft
Add teachability and a test for it.
20b26c1
@rickyloynd-microsoft
Learning from demonstration, in-progress.
9d47227
@rickyloynd-microsoft
In memory retrieval, validate insights separately rather than together.
52d4e00
@rickyloynd-microsoft
Finish learning from demonstration.
6b15777 
Seed messages with random int for variability.
@rickyloynd-microsoft
Added RecordableChatCompletionClient as a guardrail during refactoring.
a18674c
@rickyloynd-microsoft
Ran 3 evals with session recording and replay.
52e213e
@rickyloynd-microsoft
Add results to recorded sessions, including session length.
a440b0a 
Save sessions as yaml for readability.
@rickyloynd-microsoft
Use yaml file for eval settings.
cab51f1
@rickyloynd-microsoft
Simplify paths and other settings.
d91e58c
@rickyloynd-microsoft
Renamed the memory classes.
f1d7a2f
@rickyloynd-microsoft
Apprentice.
17d4c42 
Eval simplifications.
@rickyloynd-microsoft
Moved test into the evaluator, and removed eval.py's other util funct…
19654e8 
…ions.
@rickyloynd-microsoft
renaming
7aa20c1
@rickyloynd-microsoft
Rerouted calls to AgenticMemoryController through FastLearner.
83a7ddc
@rickyloynd-microsoft
Replace task_assignment_callback with AgentWrapper.
3047c1c
@rickyloynd-microsoft
Segregate files into subfolders, eval framework vs. implementation, etc.
1f20b79
@rickyloynd-microsoft
Rename FastLearner subclass to Apprentice, and import it only as spec…
de4c12b 
…ified in settings.
jackgerrits
jackgerrits requested changes Jan 29, 2025
View reviewed changes
Copy link
Member

@jackgerrits jackgerrits left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is simply too large. Please propose your changes progressively and iteratively using a separate sequence of PRs. We cannot effectively review these changes as is.

@ekzhu
Copy link
Collaborator

ekzhu commented Jan 30, 2025

@jackgerrits we will be refactoring this in the current branch first

@codecov Codecov
Copy link

codecov bot commented Feb 5, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 958 lines in your changes missing coverage. Please review.

Project coverage is 70.83%. Comparing base (c8e4ad8) to head (0298591).

Files with missing lines Patch % Lines
...-ext/src/autogen_ext/agentic_memory/page_logger.py 0.00% 259 Missing ⚠️
...en_ext/agentic_memory/agentic_memory_controller.py 0.00% 194 Missing ⚠️
...en-ext/src/autogen_ext/agentic_memory/_prompter.py 0.00% 116 Missing ⚠️
...n-ext/src/autogen_ext/agentic_memory/apprentice.py 0.00% 95 Missing ⚠️
...autogen_ext/agentic_memory/_agentic_memory_bank.py 0.00% 83 Missing ⚠️
...togen-ext/src/autogen_ext/agentic_memory/grader.py 0.00% 77 Missing ⚠️
...togen-ext/src/autogen_ext/agentic_memory/_utils.py 0.00% 65 Missing ⚠️
...togen_ext/agentic_memory/_string_similarity_map.py 0.00% 64 Missing ⚠️
...gen-ext/src/autogen_ext/agentic_memory/__init__.py 0.00% 5 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #5227      +/-   ##
==========================================
- Coverage   76.32%   70.83%   -5.49%     
==========================================
  Files         192      167      -25     
  Lines       10609    10533      -76     
  Branches      140        0     -140     
==========================================
- Hits         8097     7461     -636     
- Misses       2416     3072     +656     
+ Partials       96        0      -96
Flag Coverage Δ
unittests 70.83% <0.00%> (-5.49%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ekzhu ekzhu ekzhu left review comments

@lspinheiro lspinheiro lspinheiro left review comments

@jackgerrits jackgerrits jackgerrits requested changes

@victordibia victordibia Awaiting requested review from victordibia

@rysweet rysweet Awaiting requested review from rysweet

@husseinmozannar husseinmozannar Awaiting requested review from husseinmozannar

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rickyloynd-microsoft @ekzhu @lspinheiro @jackgerrits