aboutsummaryrefslogtreecommitdiff
path: root/internal/source/source_test.go
AgeCommit message (Collapse)Author
2023-11-07internal/source: move some tests into another packageMichael Matloob
The goal of this CL is to remove the transitive test of cmd/pkgsite on the github.com/google/go-replayers/httpreplay package. The dependency happens through the tests of the source package, so move the test out of the source package into its own package. For golang/go#61399 Change-Id: Ia0d5e4310e25ec0a034cd84f0d5cbcb06b5a75d8 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/538767 kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-31all: skip tests that can't run on wasm platformsMichael Matloob
Some because the network support isn't enough for httpreplay, and some because we need the Go command. For golang/go#61209 Change-Id: I02c6b7d10553d931c746723ce9608f39834c131c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/524797 Reviewed-by: Robert Findley <rfindley@google.com> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2023-08-29internal/source: inject *http.Client into source.NewClientMichael Matloob
This removes the dependency from package source onto ochttp. The users of source.NewClient that want an ochttp.Transport can set the transport on the *http.Client. For golang/go#61399 Change-Id: Ifb7126c482f664ee5a359f594d9324f0fd90f8b2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/523510 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2023-08-23all: remove arbitrary hard-coded timeouts in testsBryan C. Mills
If a test times out, that implies that it got stuck on something. By default, the Go testing package dumps goroutines when its own timeout is passed, which prints a goroutine dump, helping to reveal what was stuck. Adding an arbitrary timeout on top of the testing package's own timeout is, in my experience, almost always counterproductive. If the arbitrary timeout catches a real hang, it causes the test to fail instead of dumping goroutines, making it much harder to see what was stuck. On the other hand, if the timeouts are set aggressively enough to make the test fail early, they are often too aggressive for CI testing, causing flakes that then have to be triaged on an ongoing basis. On balance, the value of saving a minute or two for developers who have introduced a hang is not worth the cost of suppressing debugging information and causing flakes that have to be triaged. Fixes #61556. For #59347. Change-Id: I0263d0d9b18283470f100e5a0155818b87b5312f Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/521837 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> kokoro-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
2023-07-17internal/fetch: add two stdlib gettersMichael Matloob
This breaks out the behavior of getting the stdlib into its own getter, stdlibZipModuleGetter. Most (but not all) of the special cases in the fetch code for the stdlib are in the stdlibZipModuleGetter, which is installed on the frontend. For the local pkgsite command, we add a new builder for the goPackagesModuleGetter that loads the packages from the stdlib given a GOROOT. The stdlibZipModuleGetter is also installed for pkgsite in as a fallback if the goPackagesModuleGetter doesn't successfully fetch the stdlib module (I'm not sure if that's possible). Fixes #60114 Change-Id: Ida6a5367343643cc337c6d05e0d40095f79ca9e5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/507883 Reviewed-by: Jamal Carvalho <jamal@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> kokoro-CI: kokoro <noreply+kokoro@google.com>
2022-06-17many: re-enable staticcheck and fix problemsJonathan Amsterdam
Most of the checks were about io/ioutil. There were a couple of other minor ones. I didn't address the check for strings.Title; instead, I turned off that check globally with a staticcheck.conf file. Change-Id: I286a6894fb1fd891818ab9e451c891f52a3828fc Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/412675 Reviewed-by: Jamal Carvalho <jamal@golang.org> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2022-01-05internal/source: fix gitlab patternsJonathan Amsterdam
The source patterns for GitLab incorrectly assumed them to be identical to GitHub's. GitLab repos can have multiple path components, and the general path syntax allows for that by placing a "-" path component between the repo and reference. Fixes golang/go#49827 Change-Id: I766c303fb443b80af882931a5bfa151aa3363089 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/375281 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-09-07cmd/pkgsite,internal: serve sourceJonathan Amsterdam
- Set up a /files endpoint on the server that can serve files from fs.FS implementations. - Add source.FilesInfo, which returns a source.Info that links to /files paths. Use it to implement the SourceInfo method of the local ModuleGetters. - Add a SourceFS method to the local MethodGetters so they can tell the server how to serve their files. - Improve the tests in cmd/pkgsite to verify the source links. Fixes golang/go#47982 Change-Id: Iff42bad15c4abaf408a364e58b7c0f0dad60b40d Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347932 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2021-06-11internal/source: fix TestModuleInfoJulie Qiu
Change-Id: I4024fc8a4a3f44a62b13e05190b96ac7225dbaa5 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326776 Reviewed-by: Jonathan Amsterdam <jba@google.com> Trust: Julie Qiu <julie@golang.org>
2021-06-10internal/source: change golang.org/* reposJulie Qiu
Rather than using github.com/golang/* for the golang.org/* repos, use cs.googlesource.com/go when available. Change-Id: I88acf4c1316b160b0852ef954a515276e92ad975 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326190 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> Reviewed-by: Jamal Carvalho <jamal@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com>
2021-06-10internal/source: replace bitbucket and gitea testsJonathan Amsterdam
Replace two of the three deleted tests. The third deleted test was for git.pirl.io. That site is currently unreachable and we haven't seen a module from it since March, so I'm assuming it's offline. Fixes golang/go#46676 Change-Id: Ib0fcc8f58eeda938dbdf90199185c1424e34613a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326670 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-06-10internal/source: delete broken URLs in TestModuleInfoJulie Qiu
These tests fail when running TestModuleInfo with -record and are deleted. --- FAIL: TestModuleInfo (76.97s) source_test.go:435: Recording into testdata/TestModuleInfo.replay --- FAIL: TestModuleInfo/vanity_for_bitbucket (1.10s) source_test.go:407: https://bitbucket.org/ivucica/go-glagolitic: 401 Unauthorized --- FAIL: TestModuleInfo/gitea (1.30s) source_test.go:407: https://gitea.com/azhai/xorm: 404 Not Found Change-Id: I2ec516bb85d55a81c40ee5e38ef9aec1396a457a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/326229 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-05-12internal: change stdlib source repoJulie Qiu
The stdlib source repo is changed to https://cs.opensource.google/go/go. Fixes golang/go#44832 Change-Id: Ie8060ab1cabccbcf9ff20c0eec42c243c1feaf9c Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318873 Trust: Julie Qiu <julie@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-05-12internal/source: delete git.pirl.io testJulie Qiu
git.pirl.io/community/pirl no longer exists, so running `go test -run TestModuleInfo -record` will always return a 500 for that path. In order to update internal/source/testdata/TestModuleInfo.replay for future tests, this test is deleted. Change-Id: I8978bb8a725959896c1897e4178a5b8d6f4f51c2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318969 Trust: Julie Qiu <julie@golang.org> Run-TryBot: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2021-03-22internal/source: handle blitiri.com.ar import pathsJonathan Amsterdam
The source for these import paths is served by the git-arr web viewer, which doesn't yet handle tags. Until it does, serve the source from master. For golang/go#44607 Change-Id: I94047a887ea6f1038ae812864ad343876b74ff8e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/303609 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-02-02internal/source: trim .git suffix from repo URLsJonathan Amsterdam
Remove the ".git" from the repo URL of source templates when we know it will result in invalid URLs. The ".git" is understood by the Go command, but not by some code hosting sites. To be safe, only remove it on code hosting sites we've tested. For now, that's GitHub and GitLab. Remove the suffix in three places: 1. When we construct a SourceInfo from the repo URL and meta tags. This will fix newly processed modules. 2. When we unmarshal a SourceInfo from the DB. This will fix it during rendering for already-processed modules. It will save us from having to reprocess or backfill. 3. In the function we use in tests. Fixes golang/go#44032 Change-Id: Ic6d48d0422dc12ce8dbc4586f8d6c586f463531e Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/288612 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jamal Carvalho <jamal@golang.org>
2021-01-21internal/source: match old templates to find new onesJonathan Amsterdam
Previously, when we encountered a repo whose URL doesn't match an existing pattern, we did not generate any URL templates for it, meaning we could not render source links in the documentation. This CL uses the templates in the go-source meta tag to guess the version-aware templates that are likely to work for the repo. For golang/go#40477 Change-Id: I2d1978da5a6de1af19284233dbab9ac1ae2cb582 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/285312 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org>
2020-12-15internal/source,stdlib: change stdlib source repo to go.googlesource.comJonathan Amsterdam
Link to source files in the canonical Go repo, go.googlesource.com/go. Continue to serve raw content (for images) from the GitHub mirror, since we can't from googlesource.com. Fixes golang/go#43180 Change-Id: Iac513d6481bcdb73ae73eed40d33518ffd3a0158 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/278293 Trust: Jonathan Amsterdam <jba@google.com> Run-TryBot: Jonathan Amsterdam <jba@google.com> Reviewed-by: Julie Qiu <julie@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com>
2020-08-28internal/source: add support for dmitri.shuralyov.com/... packagesDmitri Shuralyov
Add support for a set of packages that defer to another website to display their source code. To be able to produce expected URLs, we need to add two more URL template variables: • {importPath} - Package import path ("example.com/myrepo/mypkg"). • {base} - Base name of file containing the identifier, including file extension ("file.go"). Also add a new optional Repo URL template for overriding the home page of the repository, which is something that was possible with the original go-source meta tag. Document the existing URL template variables, so that it's easier to understand how to use them. For golang/go#40477. Change-Id: I70b857155f69c5c3ed41e78daccb90153a927290 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/246239 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-07-29internal/source: support some more URLsJonathan Amsterdam
Change-Id: I23b20b711ac892a86e0a8bdceccacef6f4665191 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245380 Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-29internal/source: support gitea.* URLsJonathan Amsterdam
Treat any repo URL beginning with "gitea." as if it uses the Gitea system. Also, sanity-check templates and fix some bugs. Change-Id: Ib6c3ca028d57e05504faef4bfdf6aa4439525fb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245379 Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-28internal/source: support more code-hosting sitesJonathan Amsterdam
Suport git.fd.io, fit.pirl.io, and gitea. For golang/go#39019 Change-Id: I4e4fbf617598435803d209fc6150b7c8e56a0f72 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245259 Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-28internal/source: add sourcehut templatesJonathan Amsterdam
Add source templates for the sourcehut git hosting site (git.sr.ht). Fixes golang/go#38986. Change-Id: I581389affe3c777828189d8aa35e3910ffe1d130 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/245039 Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-21internal/source: remove HTML escapingJonathan Amsterdam
Now that the entire server uses safehtml, it is no longer necessary to do anything special with the user-provided values in this package. Every use of the Info.XXXURL methods is part of rendering a safe template, so all the escaping will happen then. Change-Id: I5e64b08d4c7b8e7fd56cf1a0bed49691f2a8ee82 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/242917 Reviewed-by: Julie Qiu <julie@golang.org>
2020-07-17internal/source: use better github raw templateJonathan Amsterdam
Replace the previous github template for raw data (raw.githubusercontent...) with a new one that is more uniform (the same as the blob template, with "raw" instead of "blob"). The main advantage is that we can drop the "repoPath" parameter to the template, which was necessary only to support the unusual GitHub URLs. Now that GitHub and GitLab templates are identical, we can drop the GitLab ones. We preserve "gitlab" in the map from kinds to templates so we can continue to unmarshal DB modules.source_info columns that use "gitlab". Change-Id: Id3d45929f0eb2c040109449d4208279cca44f079 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/243217 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-23internal/{source,frontend}: escape HTMLJonathan Amsterdam
Escape any strings that may come from outside the program and will be rendered as HTML. Also, fix some typos. Also, put args to a cmp.Diff call in the right order. Change-Id: I1fc6013e200326af8719b14c28a607fc7936e20a Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239478 Reviewed-by: Julie Qiu <julie@golang.org>
2020-06-15internal/source: really use replay client for testJonathan Amsterdam
We were creating a reply client, but not passing it in, so TestModuleInfo was actually hitting the internet. Fixes b/158850091. Change-Id: I4e981299a538750cb519c441e05805b29c7925a8 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/769682 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-05-28internal/source: update testsJonathan Amsterdam
Running with `go test -record` revealed some 404s (pages that used to exist but no longer do). I replaced them with similar pages. Change-Id: I6903229e8221923e22338bdae3f112308aecd19e Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/756267 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-04-06internal/proxy: deprecate TestVersion in favor of TestModuleJulie Qiu
proxy.TestVersion is deprecated in favor of proxy.TestModule. TestModule represents a module version used to generate testdata. Change-Id: I718045917a352af218ccebb45fa0d2875a5b9610 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709704 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-04-06internal/source: add ClientJulie Qiu
source.Client is added, which replaces the used of http.DefaultClient when fetching source code. This also allows us to set a custom timeout for tests vs when running the worker. Change-Id: I5b7b0fd32fa7a2cf836b951af1ad471751fade00 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/709198 Reviewed-by: Jonathan Amsterdam <jba@google.com>
2020-03-27internal/source: read body even on errorJonathan Amsterdam
Some sites (gonum.org in particular) return an error status (like 404) to a ?go=get=1 URL, but also serve a body with meta tags. Ignore the 404 for the http request. Fixes b/144596157. Change-Id: Ib59177b6847890975768de8fe4b623e188828f1f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/622164 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: handle raw links to the std module speciallyJonathan Amsterdam
The module directory we use to create links to the standard module's source is "src". That is only partly true, however: the .go files are there, but the README is at the repo root. This matters when constructing raw links from the README. We add a special check for this case in Info.RawURL. Note that this is a bit fragile and hacky: if we ever wanted a raw link to a source file, for example, the URL would be wrong. But the only alternative I can see now is to make everything more complicated just for this one case, which isn't worth it. Fixes b/143208537. Change-Id: Ide47b3739eaaf9f6b7e9b4631811ca3695353bb3 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/584337 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source,postgres: save source info to the DBJonathan Amsterdam
- Add a SourceInfo field to VersionInfo. - Add functions to convert a source.Info to and from JSON. - Modify postgres statements to read and write from the DB. Keep the VersionInfo.RepositoryURL field around for now to avoid messing with the frontend. Updates b/132219298, b/138752200. Change-Id: I6ebb3af3afa14fda92c2804ab2d0294742a3bf41 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/575156 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source,etl: don't make HTTP requests when testingJonathan Amsterdam
- source.ModuleInfo gains an http.Client arg - ETL tests set it to something that immediately returns an error Change-Id: I6cab563bc22424ab7037c6827309285259b7dd9c Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/574055 Reviewed-by: Robert Findley <rfindley@google.com> CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
2020-03-27internal/source: support raw content templatesJonathan Amsterdam
Add support for a URL template that can produce URLs for raw file content. To be used to generate relative links in READMEs. Change-Id: I6f6e1662825b96ddccd9dbcc4e48fbc92e7b0d82 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572879 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: support git.apache.orgJonathan Amsterdam
git.apache.org repos are hosted on GitHub. The final ".git" in the module path throws off the meta-tag matcher, so handle it specially. Change-Id: I57a8da4d4a000f0acd72945b6efa4439579e6518 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/572877 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: support gitee.com and gitlab.*Jonathan Amsterdam
Add two more patterns to our hard-coded list: - gitee.com is a GitHub look-alike, and links behave the same. - We'll assume any host of the form "gitlab.*" (e.g. "gitlab.ghn.vn") behaves like gitlab.com, which itself behaves like git. Change-Id: I4e4110699086ecd303ec4a4e5a882fb66f36c3b9 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/570673 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Robert Findley <rfindley@google.com>
2020-03-27internal/source: match go-source templates against known patternsJonathan Amsterdam
It's possible that a vanity import path's meta tags mention a known repo pattern only in the URL templates. This is what gopkg.in tags do. For example: <meta name="go-import" content="gopkg.in/yaml.v2 git https://gopkg.in/yaml.v2"> <meta name="go-source" content="gopkg.in/yaml.v2 _ https://github.com/go-yaml/yaml/tree/v2.2.3{/dir} https://github.com/go-yaml/yaml/blob/v2.2.3{/dir}/{file}#L{line}"> In this case, we can match the templates in the tag to our list of known patterns, and use the version-aware templates we get from that match instead of the ones in the tag. Here, for example, we can note that the source lives on GitHub and use the pattern that has "{commit}" instead of the hardcoded "v2.2.3", making it possible to refer to other versions of the module. Change-Id: Ia998f47ec52996eb7f71377624c638bedfe1994f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562704 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: handle versions v2 and higherJonathan Amsterdam
- Remove "vN" when constructing the module directory. - Distinguish subdirectory and branch cases by looking for a go.mod file. Fixes b/141769404. Change-Id: I20328ace5257c2102c0248e6f92bdce2debe94e5 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562701 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: support gitlab module pathsJonathan Amsterdam
gitlab.com serves source information just like github. Add it to the list of known patterns. Change-Id: I0b102d2c56c03a4ec8904f089f09ef6793db9c7f Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562177 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: handle meta tagsJonathan Amsterdam
Use the go command's meta-tag mechanism to find source code for vanity module paths. If a module path doesn't match one of our built-in patterns, make a GET request to the path and parse the go-import and go-source meta tags on the page to learn where the source lives and how to link to it. This process is slightly different than godoc.org's (as implemented in go.googlesource.com/gddo): - godoc.org uses package import paths; we use module paths instead. That can result in different behavior when a module doesn't correspond to a package and no tags are served at the module path URL, even though its package URLs do serve tags. The discovery site will not show source links for those packages. This is a genuine breaking behavior change, but we're going to make it because serving tags at the module level is the right thing to do, and we can easily fix the handful of modules where this is a problem. - godoc.org needs to fetch the code, so it needs information from the tags like the vcs, and it needs to keep the go-import and go-source tags separate (the first being for fetching, the second for linking). Since the discovery site already has the code from the zip file, we just need go-source information, using go-imports as a fallback. - The go-source tag contents don't provide a way to insert a version, so they're not usable as-is for linking to versioned code. For now, we ignore the URL templates in the tag and just try to match the repo. So a vanity URL that redirects to GitHub will work fine, since we know how to link to files in GitHub; but one that redirects to some random code-hosting site will not have source links on the discovery site, even though it might on godoc.org. Fixes b/141771975. Change-Id: Ie07bff3a77350cdaa7e45af0c7e5bce0a922e945 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/562176 Reviewed-by: Julie Qiu <julieqiu@google.com>
2020-03-27internal/source: a package for references to sourceJonathan Amsterdam
Add internal/source, which will know how to generate URLs that refer to the source code or other files inside modules. This package will be used to link to module and package source code, and also for relative links in README files. For better testing, introduce the httpreplay package. With -record, it makes HTTP requests to the actual servers and records the responses. With no flags, it replays the recordings. Fixes b/141771930. Updates b/140431331. Change-Id: I72dcd5535b8be7c6b27dd89245a048d23fd14160 Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/561217 CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com> Reviewed-by: Julie Qiu <julieqiu@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 12:45:39 +0000