| // Copyright 2024 The Go Authors. All rights reserved. |
| // Use of this source code is governed by a BSD-style |
| // license that can be found in the LICENSE file. |
| |
| // Gaby is an experimental new bot running in the Go issue tracker as [@gabyhelp], |
| // to try to help automate various mundane things that a machine can do reasonably well, |
| // as well as to try to discover new things that a machine can do reasonably well. |
| // |
| // The name gaby is short for “Go AI Bot”, because one of the purposes of the experiment |
| // is to learn what LLMs can be used for effectively, including identifying what they should |
| // not be used for. Some of the gaby functionality will involve LLMs; other functionality will not. |
| // The guiding principle is to create something that helps maintainers and that maintainers like, |
| // which means to use LLMs when they make sense and help but not when they don't. |
| // |
| // In the long term, the intention is for this code base or a successor version |
| // to take over the current functionality of “gopherbot” and become [@gopherbot], |
| // at which point the @gabyhelp account will be retired. |
| // |
| // The [GitHub Discussion] is a good place to leave feedback about @gabyhelp. |
| // |
| // # Code Overview |
| // |
| // The bot functionality is implemented in internal packages in subdirectories. |
| // This comment gives a brief tour of the structure. |
| // |
| // An explicit goal for the Gaby code base is that it run well in many different environments, |
| // ranging from a maintainer's home server or even Raspberry Pi all the way up to a |
| // hosted cloud. (At the moment, Gaby runs on a Linux server in my basement.) |
| // Due to this emphasis on portability, Gaby defines its own interfaces for all the functionality |
| // it needs from the surrounding environment and then also defines a variety of |
| // implementations of those interfaces. |
| // |
| // Another explicit goal for the Gaby code base is that it be very well tested. |
| // (See my [Go Testing talk] for more about why this is so important.) |
| // Abstracting the various external functionality into interfaces also helps make |
| // testing easier, and some packages also provide explicit testing support. |
| // |
| // The result of both these goals is that Gaby defines some basic functionality |
| // like time-ordered indexing for itself instead of relying on some specific |
| // other implementation. In the grand scheme of things, these are a small amount |
| // of code to maintain, and the benefits to both portability and testability are |
| // significant. |
| // |
| // # Testing |
| // |
| // Code interacting with services like GitHub and code running on cloud servers |
| // is typically difficult to test and therefore undertested. |
| // It is an explicit requirement in this repo to test all the code, |
| // even (and especially) when testing is difficult. |
| // |
| // A useful command to have available when working in the code is |
| // [rsc.io/uncover], which prints the package source lines not covered by a unit test. |
| // A useful invocation is: |
| // |
| // % go install rsc.io/uncover@latest |
| // % go test && go test -coverprofile=/tmp/c.out && uncover /tmp/c.out |
| // PASS |
| // ok golang.org/x/oscar/internal/related 0.239s |
| // PASS |
| // coverage: 92.2% of statements |
| // ok golang.org/x/oscar/internal/related 0.197s |
| // related.go:180,181 |
| // p.slog.Error("triage parse createdat", "CreatedAt", issue.CreatedAt, "err", err) |
| // continue |
| // related.go:203,204 |
| // p.slog.Error("triage lookup failed", "url", u) |
| // continue |
| // related.go:250,251 |
| // p.slog.Error("PostIssueComment", "issue", e.Issue, "err", err) |
| // continue |
| // % |
| // |
| // The first “go test” command checks that the test passes. |
| // The second repeats the test with coverage enabled. |
| // Running the test twice this way makes sure that any syntax or type errors |
| // reported by the compiler are reported without coverage, |
| // because coverage can mangle the error output. |
| // After both tests pass and second writes a coverage profile, |
| // running “uncover /tmp/c.out” prints the uncovered lines. |
| // |
| // In this output, there are three error paths that are untested. |
| // In general, error paths should be tested, so tests should be written |
| // to cover these lines of code. In limited cases, it may not be practical |
| // to test a certain section, such as when code is unreachable but left |
| // in case of future changes or mistaken assumptions. |
| // That part of the code can be labeled with a comment beginning |
| // “// Unreachable” or “// unreachable” (usually with explanatory text following), |
| // and then uncover will not report it. |
| // If a code section should be tested but the test is being deferred to later, |
| // that section can be labeled “// Untested” or “// untested” instead. |
| // |
| // The [golang.org/x/oscar/internal/testutil] package provides a few other testing helpers. |
| // |
| // The overview of the code now proceeds from bottom up, starting with |
| // storage and working up to the actual bot. |
| // |
| // # Secret Storage |
| // |
| // Gaby needs to manage a few secret keys used to access services. |
| // The [golang.org/x/oscar/internal/secret] package defines the interface for |
| // obtaining those secrets. The only implementations at the moment |
| // are an in-memory map and a disk-based implementation that reads |
| // $HOME/.netrc. Future implementations may include other file formats |
| // as well as cloud-based secret storage services. |
| // |
| // Secret storage is intentionally separated from the main database storage, |
| // described below. The main database should hold public data, not secrets. |
| // |
| // # Large Language Models |
| // |
| // Gaby defines the interface it expects from a large language model. |
| // |
| // The [llm.Embedder] interface abstracts an LLM that can take a collection |
| // of documents and return their vector embeddings, each of type [llm.Vector]. |
| // The two implementations to date are [golang.org/x/oscar/internal/gemini] |
| // and [golang.org/x/oscar/internal/ollama]. |
| // |
| // For tests that need an embedder but don't care about the quality of |
| // the embeddings, [llm.QuoteEmbedder] copies a prefix of the text |
| // into the vector (preserving vector unit length) in a deterministic way. |
| // This is good enough for testing functionality like vector search |
| // and simplifies tests by avoiding a dependence on a real LLM. |
| // |
| // At the moment, only the embedding interface is defined. |
| // In the future we expect to add more interfaces around text generation |
| // and tool use. |
| // |
| // # Storage |
| // |
| // As noted above, Gaby defines interfaces for all the functionality it needs |
| // from its external environment, to admit a wide variety of implementations |
| // for both execution and testing. The lowest level interface is storage, |
| // defined in [golang.org/x/oscar/internal/storage]. |
| // |
| // Gaby requires a key-value store that supports ordered traversal of key ranges |
| // and atomic batch writes up to a modest size limit (at least a few megabytes). |
| // The basic interface is [storage.DB]. |
| // [storage.MemDB] returns an in-memory implementation useful for testing. |
| // Other implementations can be put through their paces using |
| // [storage.TestDB]. |
| // |
| // The only real [storage.DB] implementation is [golang.org/x/oscar/internal/pebble], |
| // which is a [LevelDB]-derived on-disk key-value store developed and used |
| // as part of [CockroachDB]. It is a production-quality local storage implementation |
| // and maintains the database as a directory of files. |
| // |
| // In the future we plan to add an implementation using [Google Cloud Firestore], |
| // which provides a production-quality key-value lookup as a Cloud service |
| // without fixed baseline server costs. |
| // (Firestore is the successor to Google Cloud Datastore.) |
| // |
| // The [storage.DB] makes the simplifying assumption that storage never fails, |
| // or rather that if storage has failed then you'd rather crash your program than |
| // try to proceed through typically untested code paths. |
| // As such, methods like Get and Set do not return errors. |
| // They panic on failure, and clients of a DB can call the DB's Panic method |
| // to invoke the same kind of panic if they notice any corruption. |
| // It remains to be seen whether this decision is kept. |
| // |
| // In addition to the usual methods like Get, Set, and Delete, [storage.DB] defines |
| // Lock and Unlock methods that acquire and release named mutexes managed |
| // by the database layer. The purpose of these methods is to enable coordination |
| // when multiple instances of a Gaby program are running on a serverless cloud |
| // execution platform. So far Gaby has only run on an underground basement server |
| // (the opposite of cloud), so these have not been exercised much and the APIs |
| // may change. |
| // |
| // In addition to the regular database, package storage also defines [storage.VectorDB], |
| // a vector database for use with LLM embeddings. |
| // The basic operations are Set, Get, and Search. |
| // [storage.MemVectorDB] returns an in-memory implementation that |
| // stores the actual vectors in a [storage.DB] for persistence but also |
| // keeps a copy in memory and searches by comparing against all the vectors. |
| // When backed by a [storage.MemDB], this implementation is useful for testing, |
| // but when backed by a persistent database, the implementation suffices for |
| // small-scale production use (say, up to a million documents, which would |
| // require 3 GB of vectors). |
| // |
| // It is possible that the package ordering here is wrong and that VectorDB |
| // should be defined in the llm package, built on top of storage, |
| // and not the current “storage builds on llm”. |
| // |
| // # Ordered Keys |
| // |
| // Because Gaby makes minimal demands of its storage layer, |
| // any structure we want to impose must be implemented on top of it. |
| // Gaby uses the [rsc.io/ordered] encoding format to produce database keys |
| // that order in useful ways. |
| // |
| // For example, ordered.Encode("issue", 123) < ordered.Encode("issue", 1001), |
| // so that keys of this form can be used to scan through issues in numeric order. |
| // In contrast, using something like fmt.Sprintf("issue%d", n) would visit issue 1001 |
| // before issue 123 because "1001" < "123". |
| // |
| // Using this kind of encoding is common when using NoSQL key-value storage. |
| // See the [rsc.io/ordered] package for the details of the specific encoding. |
| // |
| // # Timed Storage |
| // |
| // One of the implied jobs Gaby has is to collect all the relevant information |
| // about an open source project: its issues, its code changes, its documentation, |
| // and so on. Those sources are always changing, so derived operations like |
| // adding embeddings for documents need to be able to identify what is new |
| // and what has been processed already. To enable this, Gaby implements |
| // time-stamped—or just “timed”—storage, in which a collection of key-value pairs |
| // also has a “by time” index of ((timestamp, key), no-value) pairs to make it possible |
| // to scan only the key-value pairs modified after the previous scan. |
| // This kind of incremental scan only has to remember the last timestamp processed |
| // and then start an ordered key range scan just after that timestamp. |
| // |
| // This convention is implemented by [golang.org/x/oscar/internal/timed], along with |
| // a [timed.Watcher] that formalizes the incremental scan pattern. |
| // |
| // # Document Storage |
| // |
| // Various packages take care of downloading state from issue trackers and the like, |
| // but then all that state needs to be unified into a common document format that |
| // can be indexed and searched. That document format is defined by |
| // [golang.org/x/oscar/internal/docs]. A document consists of an ID (conventionally a URL), |
| // a document title, and document text. Documents are stored using timed storage, |
| // enabling incremental processing of newly added documents . |
| // |
| // # Document Embedding |
| // |
| // The next stop for any new document is embedding it into a vector and storing |
| // that vector in a vector database. The [golang.org/x/oscar/internal/embeddocs] package |
| // does this, and there is very little to it, given the abstractions of a document store |
| // with incremental scanning, an LLM embedder, and a vector database, all of which |
| // are provided by other packages. |
| // |
| // # HTTP Record and Replay |
| // |
| // None of the packages mentioned so far involve network operations, but the |
| // next few do. It is important to test those but also equally important not to |
| // depend on external network services in the tests. Instead, the package |
| // [golang.org/x/oscar/internal/httprr] provides an HTTP record/replay system specifically |
| // designed to help testing. It can be run once in a mode that does use external |
| // network servers and records the HTTP exchanges, but by default tests look up |
| // the expected responses in the previously recorded log, replaying those responses. |
| // |
| // The result is that code making HTTP request can be tested with real server |
| // traffic once and then re-tested with recordings of that traffic afterward. |
| // This avoids having to write entire fakes of services but also avoids needing |
| // the services to stay available in order for tests to pass. It also typically makes |
| // the tests much faster than using the real servers. |
| // |
| // # GitHub Interactions |
| // |
| // Gaby uses GitHub in two main ways. First, it downloads an entire copy of the |
| // issue tracker state, with incremental updates, into timed storage. |
| // Second, it performs actions in the issue tracker, like editing issues or comments, |
| // applying labels, or posting new comments. These operations are provided by |
| // [golang.org/x/oscar/internal/github]. |
| // |
| // Gaby downloads the issue tracker state using GitHub's REST API, which makes |
| // incremental updating very easy but does not provide access to a few newer features |
| // such as project boards and discussions, which are only available in the GraphQL API. |
| // Sync'ing using the GraphQL API is left for future work: there is enough data available |
| // from the REST API that for now we can focus on what to do with that data and not |
| // that a few newer GitHub features are missing. |
| // |
| // The github package provides two important aids for testing. For issue tracker state, |
| // it also allows loading issue data from a simple text-based issue description, avoiding |
| // any actual GitHub use at all and making it easier to modify the test data. |
| // For issue tracker actions, the github package defaults in tests to not actually making |
| // changes, instead diverting edits into an in-memory log. Tests can then check the log |
| // to see whether the right edits were requested. |
| // |
| // The [golang.org/x/oscar/internal/githubdocs] package takes care of adding content from |
| // the downloaded GitHub state into the general document store. |
| // Currently the only GitHub-derived documents are one document per issue, |
| // consisting of the issue title and body. It may be worth experimenting with |
| // incorporating issue comments in some way, although they bring with them |
| // a significant amount of potential noise. |
| // |
| // # Gerrit Interactions |
| // |
| // Gaby will need to download and store Gerrit state into the database and then |
| // derive documents from it. That code has not yet been written, although |
| // [rsc.io/gerrit/reviewdb] provides a basic version that can be adapted. |
| // |
| // # Web Crawling |
| // |
| // Gaby will also need to download and store project documentation into the |
| // database and derive documents from it corresponding to cutting the page |
| // at each heading. That code has been written but is not yet tested well enough |
| // to commit. It will be added later. |
| // |
| // # Fixing Comments |
| // |
| // The simplest job Gaby has is to go around fixing new comments, including |
| // issue descriptions (which look like comments but are a different kind of GitHub data). |
| // The [golang.org/x/oscar/internal/commentfix] package implements this, |
| // watching GitHub state incrementally and applying a few kinds of rewrite rules |
| // to each new comment or issue body. |
| // The commentfix package allows automatically editing text, automatically editing URLs, |
| // and automatically hyperlinking text. |
| // |
| // # Finding Related Issues and Documents |
| // |
| // The next job Gaby has is to respond to new issues with related issues and documents. |
| // The [golang.org/x/oscar/internal/related] package implements this, |
| // watching GitHub state incrementally for new issues, filtering out ones that should be ignored, |
| // and then finding related issues and documents and posting a list. |
| // |
| // This package was originally intended to identify and automatically close duplicates, |
| // but the difference between a duplicate and a very similar or not-quite-fixed issue |
| // is too difficult a judgement to make for an LLM. Even so, the act of bringing forward |
| // related context that may have been forgotten or never known by the people reading |
| // the issue has turned out to be incredibly helpful. |
| // |
| // # Main Loop |
| // |
| // All of these pieces are put together in the main program, this package, [golang.org/x/oscar]. |
| // The actual main package has no tests yet but is also incredibly straightforward. |
| // It does need tests, but we also need to identify ways that the hard-coded policies |
| // in the package can be lifted out into data that a natural language interface can |
| // manipulate. For example the current policy choices in package main amount to: |
| // |
| // cf := commentfix.New(lg, gh, "gerritlinks") |
| // cf.EnableProject("golang/go") |
| // cf.EnableEdits() |
| // cf.AutoLink(`\bCL ([0-9]+)\b`, "https://go.dev/cl/$1") |
| // cf.ReplaceURL(`\Qhttps://go-review.git.corp.google.com/\E`, "https://go-review.googlesource.com/") |
| // |
| // rp := related.New(lg, db, gh, vdb, dc, "related") |
| // rp.EnableProject("golang/go") |
| // rp.EnablePosts() |
| // rp.SkipBodyContains("— [watchflakes](https://go.dev/wiki/Watchflakes)") |
| // rp.SkipTitlePrefix("x/tools/gopls: release version v") |
| // rp.SkipTitleSuffix(" backport]") |
| // |
| // These could be stored somewhere as data and manipulated and added to by the LLM |
| // in response to prompts from maintainers. And other features could be added and |
| // configured in a similar way. Exactly how to do this is an important thing to learn in |
| // future experimentation. |
| // |
| // # Future Work and Structure |
| // |
| // As mentioned above, the two jobs Gaby does already are both fairly simple and straightforward. |
| // It seems like a general approach that should work well is well-written, well-tested deterministic |
| // traditional functionality such as the comment fixer and related-docs poster, configured by |
| // LLMs in response to specific directions or eventually higher-level goals specified by project |
| // maintainers. Other functionality that is worth exploring is rules for automatically labeling issues, |
| // rules for identifying issues or CLs that need to be pinged, rules for identifying CLs that need |
| // maintainer attention or that need submitting, and so on. Another stretch goal might be to identify |
| // when an issue needs more information and ask for that information. Of course, it would be very |
| // important not to ask for information that is already present or irrelevant, so getting that right would |
| // be a very high bar. There is no guarantee that today's LLMs work well enough to build a useful |
| // version of that. |
| // |
| // Another important area of future work will be running Gaby on top of cloud databases |
| // and then moving Gaby's own execution into the cloud. Getting it a server with a URL will |
| // enable GitHub callbacks instead of the current 2-minute polling loop, which will enable |
| // interactive conversations with Gaby. |
| // |
| // Overall, we believe that there are a few good ideas for ways that LLM-based bots can help |
| // make project maintainers' jobs easier and less monotonous, and they are waiting to be found. |
| // There are also many bad ideas, and they must be filtered out. Understanding the difference |
| // will take significant care, thought, and experimentation. We have work to do. |
| // |
| // [@gabyhelp]: https://github.com/gabyhelp |
| // [@gopherbot]: https://github.com/gopherbot |
| // [GitHub Discussion]: https://github.com/golang/go/discussions/67901 |
| // [LevelDB]: https://github.com/google/leveldb |
| // [CockroachDB]: https://github.com/cockroachdb/cockroach |
| // [Google Cloud Firestore]: https://cloud.google.com/firestore |
| // [Go Testing talk]: https://research.swtch.com/testing |
| package main |
