Teachability vs codebase

[kfsone]

Off-the-cuff, teachability sounds like a game changer for code-assisting LMs, but what are the limitations? We have a proprietary codebase with hundreds of 1st-party C++ cmake-library targets, and hundreds more 3rd party libraries.

In theory it seems like I could set up a multi-group autogen with the goal of studying the codebase and being able to detect replicated functionalities, redundant implementations, detached spaghetti code, and answer "how do I?" questions.

But is that stretching the practical limitations of memory? Is this an undertaking that would be better achieved by extending training of a model like Llama2 or something with the codebase?

[sonichi]

@rickyloynd-microsoft @gagb might be interested in this topic.

[qingyun-wu]

@skzhang1 might be interested in this as well.

[gagb]

This is a great questions! Putting on my researcher hat -- its an open question and we'd probably have to implement both and measure empirical performance!

[kfsone]

As best I understand it, you probably don't want to train the LLM itself on your source code repository, because it's a deceptively different kind of pattern. I mean, if you took a c++ codebase from 20 years ago you would just be training a buffer-overrun model.

For a shareable frame of reference, consider the state of a python virtual environment after installing jupyter to access 'ipython'. Due to nested depdencies, you now have different async packages, networking tools, xml parsers even; and then there's all the times each of the 40+ dependencies have an accessible if not intentionally exposed method/helper that does something very similar to 2-or-more of the other dependencies; I mean, you have 4 different lru implementations, one async.

Somewhere in there, ipython has the capability to print colorized text, but I don't know what/what, so I fall back to also installing rich and using that. And, because nobody in the training data said "you could just use this built-in facility", the laziness pattern is in most of the LLM/LMs I tried this thought out on.

[USER]> I'm trying to print colorized text from python, so I've been hand coding ascii escape sequences. I've tested it in ipython but it feels clutzy.
[AI]> ... !pip install ...  <- ! prefix because of the ipython context
[USER]> I'm doing this in my [i]python startup script  <-- expect this to be a hint, but 4+ regenerates with each LM never diverted
[AI]> ... add colorama to your requirements.txt ...

It's not that it didn't ever say: Well, in ipython, you can do this - someone appears to have replied with that sometime, codellama instruct q6_k did it on the 3rd regenerate, but was unfortunately also wrong.