Migrate package download redirect to using V4 signed urls

[jonasfj]

With this buckets do not have to be public anymore.

Granted we still need to change GCLB configuration.

[jonasfj]

fyi, I deployed this to staging-sigurdm

[sigurdm]

Two things here:

1. Encapsulation: PackageBackend already has this._exportedApiBucket injected. Consider using _exportedApiBucket.bucketName instead of activeConfiguration.exportedApiBucketName!.

2. IAM permissions: Testing this on staging-sigurdm (/api/archives/cancellable-2.3.0.tar.gz) currently fails with HTTP 403 AccessDenied:

   package-uploader-signer@dartlang-pub-dev.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object.
   

   GCS verifies that the signer identity in X-Goog-Credential actually has read permissions on the object. package-uploader-signer needs roles/storage.objectViewer on dartlang-pub-dev-exported-api (and prod dartlang-pub-exported-api). We should also update the doc comment on Configuration.uploadSignerServiceAccount to mention this requirement.

[jonasfj]

🙈 I thought I had tested this? maybe bucket permission changes hadn't gone through.

But yes, you're write package-uploader-signer doesn't have sufficient permissions, and giving them to it would be a bit of a hack.

[sigurdm]

What happens when package versions have build metadata containing + (e.g. foo-1.0.0+1.tar.gz)?

In the GCS V4 signing spec:

When defining the resource path, you must percent encode the following reserved characters: ?!#$&'()*+,:;@[]"

Here canonicalUri keeps a literal + and passes it both to the canonical request and Uri(path: canonicalUri). If a client, proxy, or CDN encodes + as %2B when following the redirect, GCS calculates the canonical request with %2B and rejects the request with 403 SignatureDoesNotMatch (tested and confirmed against GCS).

Reserved characters like + should be percent-encoded in both the canonical URI and the returned Uri.

[sigurdm]

The embedded \n in 'host:storage.googleapis.com\n' works because it creates the required blank line after canonical headers when joined with \n, but it looks like a typo.

Consider separating it explicitly so the structure is obvious:

final canonicalHeaders = 'host:storage.googleapis.com\n';
final canonicalRequest = [
  'GET',
  canonicalUri,
  canonicalQueryString,
  canonicalHeaders,
  'host',
  'UNSIGNED-PAYLOAD',
].join('\n');

[sigurdm]

Because FakeUploadSignerService completely bypasses UploadSignerService.buildDownloadUrl, none of our unit/integration tests actually execute the signing algorithm.

Could we add a unit test for UploadSignerService.buildDownloadUrl (using a mock sign implementation) to verify the date format, query string sorting, canonical request hashing, and URL encoding against known test vectors?

[sigurdm]

Is there a reason to keep the lifetime at 15 minutes?

GCS V4 signed URLs support a lifetime of up to 7 days (Duration(days: 7) / 604,800 seconds). Because pub package archives are public open-source code rather than confidential user data, a longer lifetime (e.g. up to 7 days) would allow us to cache the signed URLs in Redis or memory for days at a time.

That would eliminate almost all outbound signBlob IAM RPCs and the associated ~100ms latency on download redirects. Note that even with a multi-day signature, if a package version is deleted or moderated, GCS returns 404 NoSuchKey immediately.

[sigurdm]

An alternative to calling iam.signBlob over HTTP for download URLs is to sign locally using a GCS HMAC key (GOOG4-HMAC-SHA256).

GCS natively supports V4 signed URLs using HMAC keys associated with service accounts (docs).

Instead of awaiting the iam.signBlob RPC (which adds 50–150ms of latency per redirect), we could:

1. Create an HMAC key for the service account (gcloud storage hmac create <service-account>).
2. Store the secret in Secret Manager.
3. Derive the signing key and calculate the signature locally in Dart using package:crypto:

final kDate = Hmac(sha256, utf8.encode('GOOG4$secret')).convert(utf8.encode(date)).bytes;
final kRegion = Hmac(sha256, kDate).convert(utf8.encode('auto')).bytes;
final kService = Hmac(sha256, kRegion).convert(utf8.encode('storage')).bytes;
final signingKey = Hmac(sha256, kService).convert(utf8.encode('goog4_request')).bytes;

final signature = hex.encode(Hmac(sha256, signingKey).convert(utf8.encode(stringToSign)).bytes);

The Canonical Request and String-To-Sign construction remain identical; only X-Goog-Algorithm becomes GOOG4-HMAC-SHA256 and X-Goog-Credential uses the HMAC accessId.

This would reduce the signing latency to < 0.1ms with zero network RPCs or IAM quotas. Worth considering if archive redirects are going to be served by the backend.