Commit graph

4361 commits

Author SHA1 Message Date
Pavlo Tkach
8ce47b9e0e angular version update 15.1.0 -> 15.2.2 (#1967) 2023-03-16 11:56:38 -04:00
gbrodman
cd0703c815 Allow for multiple service accounts in authentication (#1963)
When submitting tasks to Cloud Tasks, we will use the built-in OIDC
authentication which runs under the default service account (not the
cloud scheduler service account). We want either to work for app-level
auth.
2023-03-15 10:20:58 -04:00
gbrodman
8ab2252272 Move CloudTasksUtils to core/ project (#1956)
This does nothing for now, but in the future this will allow us to refer
to the RegistryConfig and/or Service objects from the core project. This
will be necessary when changing CloudTasksUtils to not use the AppEngine
built-in connection (it will need to use a standard HTTP request
instead).
2023-03-14 15:15:05 -04:00
Lai Jiang
8a4252186a Rename process_time column in DnsRefreshRequest (#1962)
Make it explicit that this is the last process time, not a scheduled
future process time.
2023-03-14 14:03:12 -04:00
Pavlo Tkach
59f457b2d7 Use builder image as a base for schema-deployer and schema-verifier (#1955) 2023-03-13 15:37:02 -04:00
Pavlo Tkach
d996ef974c Expand ID Token Auth verifier to catch all exceptions (#1960) 2023-03-13 12:12:47 -04:00
Pavlo Tkach
de6d73930c Restore resaveAllEppResourcesPipeline as a cloud task (#1953) 2023-03-13 10:44:25 -04:00
sarahcaseybot
04315ee940 Add allowedEppActions field to AllocationToken (#1957) 2023-03-10 14:14:47 -05:00
Pavlo Tkach
7146260a08 Remove app engine deps from Lock (#1910) 2023-03-09 10:47:48 -05:00
Lai Jiang
09d19d996d Drop the indexs and columns for dns_refresh_request_time (#1949) 2023-03-09 10:29:31 -05:00
gbrodman
253aa7d9ad Add create/delete/update commands for User objects (#1936)
This also includes the change of allowing the Java User object to have a
null GAIA ID (when creating user objects, we don't know what the GAIA ID
is).
2023-03-07 17:18:48 -05:00
Ben McIlwain
93de7d689f Double the number of frontend instances from 12 to 24 (#1954)
It seems like we're hitting App Engine capacity issues resulting in actual pages
now (for whatever reason, but likely one customer), and we obviously don't want
that.
2023-03-06 16:04:28 -05:00
Pavlo Tkach
f615e88ff6 Move App Engine cron jobs to cloud scheduler (#1939) 2023-03-01 13:40:56 -05:00
Lai Jiang
7f3a43f70d Remove dnsRefreshRequestTime from EppResources (#1943)
We have decided to use a separate table (#1940) to track DNS refresh requests
due to performance reasons.

See: go/registry-pull-queue-redesign
2023-03-01 13:40:30 -05:00
Lai Jiang
9137978083 Add a process time column to DnsRefreshRequest (#1948)
The value of the column would be set to START_OF_TIME for new entries.
Every time a row is read, the value is updated to the current time. This
allows concurrent reads to not repeatedly read the same entry that has the
earliest request time, because they would only look for rows that have a value
of process time that is before current time - some padding time.

This basically fulfills the same function that LEASE_PADDING gives us
when using a pull queue, whereas a task would be leased for a certain
time, during which time they would not be leased by anyone else.

See: https://cs.opensource.google/nomulus/nomulus/+/master:core/src/main/java/google/registry/dns/ReadDnsQueueAction.java;l=99?q=readdnsqueue&ss=nomulus%2Fnomulus
2023-02-28 16:52:02 -05:00
sarahcaseybot
9a212314ca Allow incorrect fee extensions on domain creates with default tokens (#1927)
* Modify fee extension to accept larger costs on creates with default tokens

* Add tests

* Add some comments to tests
2023-02-28 14:24:03 -05:00
Lai Jiang
c07728dab8 Add support for Nordn upload without using pull queues. (#1925)
This PR adds an alternative method to upload Lordn to Nordn server without
using App Engine pull queue. A new database migration stage is added to control
whether a new task is scheduled with the old or new method. The
NordnUploadAction is configured to process both kind of tasks. Once the tasks
scheduled for the old tasks are all processed, we can start using the
new method exclusively.

See: go/registry-pull-queue-redesign
2023-02-28 12:57:27 -05:00
Lai Jiang
33f9bc30b7 Ignore invalid old CRL when performing update. (#1946)
There is no point comparing the old CRL to the new ones when the old one
is invalid. This could happen when the CA cert rotates, after which the
old CRL stop being valid as it fails signature verification against the
new cert.

This change will allow us to keep updating the CRL after a CA rotation without
having to manually delete the old CRL from the database.

See b/270983553.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1946)
<!-- Reviewable:end -->
2023-02-28 10:00:18 -05:00
Lai Jiang
16836475cd Update kythe vnames mapping (#1944) 2023-02-27 17:09:57 -05:00
Ben McIlwain
b9045cce86 Remove "letter G with stroke" from Extended Latin IDN table (#1938)
ICANN doesn't like this character because it's confusable with a normal G (the
stroke tends to get lost in the visual clutter of the descender), and .com's
Extended Latin table doesn't use it either. Best to get rid of it.
2023-02-23 16:27:15 -05:00
Lai Jiang
556bd09b3b Add allowed license for json (#1942)
For some reason `./gradlew clean build` on master is failing for me on
multiple machines due to a new org.json:json version triggering license
violations, even though the lock files are not changing.

Note that the old versions are still present because if I remove
"The JSON license", which the old versions use, the check also fails...
2023-02-23 11:37:31 -05:00
Lai Jiang
63096c68a8 Add a table to record EPP resources needing DNS refresh (#1940) 2023-02-22 14:18:28 -05:00
Lai Jiang
bac9a25800 Change default beam job region (#1937)
For reasons that I cannot explain, the same expand recurring billing
event pipeline would fail in us-east1 but succeed in us-central1.

See:

https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-02-09_14_52_24-162498476138221714;graphView=0?project=domain-registry

https://pantheon.corp.google.com/dataflow/jobs/us-east1/2023-02-09_14_26_07-4564782062878841960;graphView=1?project=domain-registry

Also improved how the accuracy of the metrics:

It is observed that both counters are consistently higher for the same
start and end times when running in dry run mode. There is no way to
test for consistency when not running in dry run, for obviously reasons.

I can make the recurrings in scope counter consistent by not updating it
in a side-effect-causing transaction, but there is no way around the
other counter. It can only be trusted when running in dry run mode,
unfortunately.
2023-02-13 15:57:32 -05:00
gbrodman
b709a0ce48 Add a frontend endpoint for retrieving a domain in JSON form (#1916)
We might (likely will) modify some of the fiddly bits around this (maybe
the GSON serialization, where we do the actual authorization, etc) but
this should be a decent basic shell structure for endpoints that the new
registrar console can call to retrieve JSON results.
2023-02-09 15:09:42 -05:00
gbrodman
e26e5adf5c Generate and use an IAP-enabled ID token in the proxy (#1926)
This is only generated and used if "iapClientId" is set in the proxy
config. If so, we use code similar to
https://cloud.google.com/iap/docs/authentication-howto#obtaining_an_oidc_token_for_the_default_service_account
to generate an ID token that is valid for IAP. We set the token on the
Proxy-Authorization header so that we can keep using the pre-existing
access token as well -- IAP allows for us to use either the
Authorization header or the Proxy-Authorization header.
2023-02-09 14:50:35 -05:00
gbrodman
de1f56393d Allow null GAIA IDs for User objects (#1933)
We were under the mistaken impression before that there was a reliable
way to, out-of-band, get a GAIA ID for a particular email address.
Unfortunately, that isn't the case (at least, not in a scalable way or
one that support agents could use). As a result, we have to allow null
GAIA IDs in the database.

When we (or the support team) create new users, we will only specify the
email address and not the GAIA ID. Then, when the user logs in for the
first time, we will have the GAIA ID from the provided ID token, and we
can populate it then.
2023-02-08 16:10:34 -05:00
Lai Jiang
db9b5e48a2 Install procps in schema-deployer image (#1934)
It turns out this one uses pgrep and pkill as well, go figure...

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1934)
<!-- Reviewable:end -->
2023-02-08 09:59:47 -05:00
Lai Jiang
3532213d4c Update TMCH root certificate (#1918)
See b/260945047.

Also refactored the corresponding tests, which should future updates easier.

This change should be deployed at or around 2023-02-15T16:00:00Z.
2023-02-06 22:39:54 -05:00
Lai Jiang
80e129907e Install procps (#1932)
The schema verifier script needs pgrep and pkill, which do not come with
Debian.
2023-02-06 19:45:04 -05:00
Lai Jiang
4fa3556eee Install curl (#1931)
Tested by running "docker build .".
2023-02-06 16:45:52 -05:00
dependabot[bot]
f87588755c Bump http-cache-semantics from 4.1.0 to 4.1.1 in /console-webapp (#1929)
Bumps [http-cache-semantics](https://github.com/kornelski/http-cache-semantics) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/kornelski/http-cache-semantics/releases)
- [Commits](https://github.com/kornelski/http-cache-semantics/compare/v4.1.0...v4.1.1)

---
updated-dependencies:
- dependency-name: http-cache-semantics
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 13:22:50 -05:00
Lai Jiang
131bf6ac5c Change base image for schema-verifier and schema-deployer (#1930)
Ubuntu 18.04 is entering EOL and the Cloud Build jobs are failing,
seemingly due to connection error to 18.04 repos:

https://pantheon.corp.google.com/cloud-build/builds;region=global/126a7c90-4322-41f1-ba1c-a10e38a32dab;step=5?project=domain-registry-dev

We use Debian 10 for the main builder, so it's better to keep everything
on the same schedule:

https://cs.opensource.google/nomulus/nomulus/+/master:release/builder/Dockerfile

Debian 10 is supported till June 2024:

https://wiki.debian.org/LTS
2023-02-06 13:09:37 -05:00
Lai Jiang
706cf69b7f Fix expand recurring billing event pipeline (#1928) 2023-02-06 11:33:57 -05:00
sarahcaseybot
c6e77276b6 Modify DomainCreateFlow to check for an applicable defaultPromoToken (#1904)
* Modify DomainCreateFlow to check for an applicable defaultPromoToken

* Add handling for deleted tokens

* Change cache to allocation token cache

* Abstract away cache methods

* Use AllocationToken.getAll in create flow

* Filter out empty tokens
2023-02-01 14:53:51 -05:00
dependabot[bot]
5a1d84a27b Bump ua-parser-js from 0.7.31 to 0.7.33 in /console-webapp (#1924)
Bumps [ua-parser-js](https://github.com/faisalman/ua-parser-js) from 0.7.31 to 0.7.33.
- [Release notes](https://github.com/faisalman/ua-parser-js/releases)
- [Changelog](https://github.com/faisalman/ua-parser-js/blob/master/changelog.md)
- [Commits](https://github.com/faisalman/ua-parser-js/compare/0.7.31...0.7.33)

---
updated-dependencies:
- dependency-name: ua-parser-js
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lai Jiang <jianglai@google.com>
2023-01-31 14:52:26 -05:00
sarahcaseybot
f1cbab258f Prevent ending package tokens with active domains (#1919)
* Prevent ending package tokens with active domains

* Fix bad formatting in comments

* Fix lots of nits
2023-01-30 16:13:23 -05:00
Lai Jiang
946b683906 Remove @IdAllocation annotation from repoId (#1923)
This annotation only works for Long or long field.
2023-01-30 15:40:40 -05:00
gbrodman
bd66b3cef1 Use the Proxy-Authorization header when using nomulus + IAP (#1921) 2023-01-26 15:16:32 -05:00
Lai Jiang
ab146c4bf8 Add fields needed to implement pull queue alternative (#1915) 2023-01-25 15:26:00 -05:00
Pavlo Tkach
5a3aa9fa52 Add a condition update precaution to validateNewState (#1920) 2023-01-25 14:53:12 -05:00
Weimin Yu
c71a3312ab Upgrade to Gradle 7.3.2 (#1922)
This is an 'easy' upgrade that requires a minor change in
common/build.gradle and the removal of an unnecessary import in buildSrc.

Gradle 7.4 and above has breaking changes that break the latest nebula lint plugin. We may have to wait a while.
2023-01-25 12:47:35 -05:00
Lai Jiang
8088efb25c Update ExpandRecurringBillingEventsAction to use the beam pipeline (#1907)
Due to the way the beam pipeline is designed, it will expand an
recurring billing event when its event time is in scope for expansion,
instead of billing time. This means that the one time will be generated
45 days earlier. This would negate the need to check if the expansion is
finished when generating monthly invoices.

We will need to backfill the past 45 days of onetimes before the new
code is deployed. As an illustration, with the old code, a cursor time
of 2023-01-17 means that all auto-renewals whose billing time is before
2023-01-17 were created, which corresponds to an effective cursor time
of 2022-12-03 (45 days before 2023-01-17) for event time. This cursor
will need to be brought to 2023-01-17 to ensure that there is no gap in
generated event times when switching to use the new code.
2023-01-23 19:08:04 -05:00
Pavlo Tkach
5210ac0419 Remove not null constraint request_log_id column (#1917) 2023-01-23 09:37:20 -05:00
Lai Jiang
4250a9d165 Add columns needed to implement pull queue alternative (#1914) 2023-01-20 14:17:06 -05:00
Lai Jiang
79d4be7e99 Delete TaskQueueUtils (#1908)
For push queues, use CloudTasksUtils. Pull queues for now directly calls
the GAE task queue APIs. The usage of pull queues will be soon replaced.
2023-01-19 14:45:18 -05:00
Lai Jiang
e41fd7877e Remove datastore related code (#1906) 2023-01-19 14:44:11 -05:00
Lai Jiang
913edb23ee Do not deploy datastore index file (#1913)
The index was deleted in #1905.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1913)
<!-- Reviewable:end -->
2023-01-18 16:31:35 -05:00
Lai Jiang
408fff4fc1 Use a fake instance id in metric (#1912)
Currently we synthesize a instance id which requires the use of App
Engine Module API. Given that we only have one version of code running
at one time, and HTTP is stateless, there is no point tracking exactly
which GAE "instance" is. We do lose information on which service (default,
backend, etc) is writing the metric, but that does not seem very
important.

Using a constant fake instance ID allows us to get rid of another GAE
dependency.
2023-01-18 16:24:59 -05:00
Lai Jiang
f65b6ece5d Fix flaky tests with TaskQueueExtension (#1909)
The temporary queue.xml file is not deleted in the afterEach() method,
likely causing some flaky tests that we saw due to overwriting of the
file by concurrent tests.
2023-01-18 12:04:47 -05:00
sarahcaseybot
4067f7c3e4 Only email support for package non-compliance (#1900)
* Only email support for package non-compliance

* Fix import

* Always use longs
2023-01-17 14:22:15 -05:00