Commit graph

4668 commits

Author SHA1 Message Date
Pavlo Tkach
c216c874b4
Remove app engine deps from Lock flyway change (#1911) 2023-03-20 12:25:12 -04:00
Pavlo Tkach
0ab9471c8d
Make cloud scheduler deployment part of gradle deploy (alpha, qa and crash only) (#1969) 2023-03-20 11:10:00 -04:00
sarahcaseybot
d482754f66
Implement default tokens for the fee extension in domain check flow (#1950)
* Implement default tokens for the fee extension in domain check

* Add test for expired token

* Add test for alloc token and default token

* Fix formatting

* Always check for default tokens

* Change transaction time to passed in DateTime
2023-03-17 15:41:17 -04:00
sarahcaseybot
fe086b43f5
Add TTL columns to the Tld table (#1964)
* Add TTL columns to Tld table

* Change A and AAAA records column name
2023-03-17 11:54:14 -04:00
Lai Jiang
95f1bca3fb
Remove Nordn pull queue code (#1966)
The SQL-based flow is verified to work on production.
2023-03-16 17:37:48 -04:00
sarahcaseybot
178a2323d9
Add allowedEppActions to AllocationToken Java classes (#1958)
* Add allowedEppActions field to AllocationToken Java class and converter

* Add getter and setter
2023-03-16 15:45:34 -04:00
Lai Jiang
a44aa1378f
Create a DnsRefreshRequest entity backed by the corresponding table (#1941)
Also adds a DnsUtils class to deal with adding, polling, and removing
DNS refresh requests (only adding is implemented for now). The class
also takes care of choosing which mechanism to use (pull queue vs. SQL)
based on the current time and the database migration schedule map.
2023-03-16 13:02:20 -04:00
Pavlo Tkach
d0f625f70e
angular version update 15.1.0 -> 15.2.2 (#1967) 2023-03-16 11:56:38 -04:00
gbrodman
fb59874234
Allow for multiple service accounts in authentication (#1963)
When submitting tasks to Cloud Tasks, we will use the built-in OIDC
authentication which runs under the default service account (not the
cloud scheduler service account). We want either to work for app-level
auth.
2023-03-15 10:20:58 -04:00
gbrodman
b6083e227f
Move CloudTasksUtils to core/ project (#1956)
This does nothing for now, but in the future this will allow us to refer
to the RegistryConfig and/or Service objects from the core project. This
will be necessary when changing CloudTasksUtils to not use the AppEngine
built-in connection (it will need to use a standard HTTP request
instead).
2023-03-14 15:15:05 -04:00
Lai Jiang
5805b6859e
Rename process_time column in DnsRefreshRequest (#1962)
Make it explicit that this is the last process time, not a scheduled
future process time.
2023-03-14 14:03:12 -04:00
Pavlo Tkach
3108e8a871
Use builder image as a base for schema-deployer and schema-verifier (#1955) 2023-03-13 15:37:02 -04:00
Pavlo Tkach
ec142caf9c
Expand ID Token Auth verifier to catch all exceptions (#1960) 2023-03-13 12:12:47 -04:00
Pavlo Tkach
e60ad58098
Restore resaveAllEppResourcesPipeline as a cloud task (#1953) 2023-03-13 10:44:25 -04:00
sarahcaseybot
83e9e7fb5c
Add allowedEppActions field to AllocationToken (#1957) 2023-03-10 14:14:47 -05:00
Pavlo Tkach
438c523fcb
Remove app engine deps from Lock (#1910) 2023-03-09 10:47:48 -05:00
Lai Jiang
025a2faff2
Drop the indexs and columns for dns_refresh_request_time (#1949) 2023-03-09 10:29:31 -05:00
gbrodman
fd822dd333
Add create/delete/update commands for User objects (#1936)
This also includes the change of allowing the Java User object to have a
null GAIA ID (when creating user objects, we don't know what the GAIA ID
is).
2023-03-07 17:18:48 -05:00
Ben McIlwain
9b93749d43
Double the number of frontend instances from 12 to 24 (#1954)
It seems like we're hitting App Engine capacity issues resulting in actual pages
now (for whatever reason, but likely one customer), and we obviously don't want
that.
2023-03-06 16:04:28 -05:00
Pavlo Tkach
71a8579ece
Move App Engine cron jobs to cloud scheduler (#1939) 2023-03-01 13:40:56 -05:00
Lai Jiang
cda51f13dc
Remove dnsRefreshRequestTime from EppResources (#1943)
We have decided to use a separate table (#1940) to track DNS refresh requests
due to performance reasons.

See: go/registry-pull-queue-redesign
2023-03-01 13:40:30 -05:00
Lai Jiang
1de5b5dcc1
Add a process time column to DnsRefreshRequest (#1948)
The value of the column would be set to START_OF_TIME for new entries.
Every time a row is read, the value is updated to the current time. This
allows concurrent reads to not repeatedly read the same entry that has the
earliest request time, because they would only look for rows that have a value
of process time that is before current time - some padding time.

This basically fulfills the same function that LEASE_PADDING gives us
when using a pull queue, whereas a task would be leased for a certain
time, during which time they would not be leased by anyone else.

See: https://cs.opensource.google/nomulus/nomulus/+/master:core/src/main/java/google/registry/dns/ReadDnsQueueAction.java;l=99?q=readdnsqueue&ss=nomulus%2Fnomulus
2023-02-28 16:52:02 -05:00
sarahcaseybot
32279e42e4
Allow incorrect fee extensions on domain creates with default tokens (#1927)
* Modify fee extension to accept larger costs on creates with default tokens

* Add tests

* Add some comments to tests
2023-02-28 14:24:03 -05:00
Lai Jiang
ba0f90bdaf
Add support for Nordn upload without using pull queues. (#1925)
This PR adds an alternative method to upload Lordn to Nordn server without
using App Engine pull queue. A new database migration stage is added to control
whether a new task is scheduled with the old or new method. The
NordnUploadAction is configured to process both kind of tasks. Once the tasks
scheduled for the old tasks are all processed, we can start using the
new method exclusively.

See: go/registry-pull-queue-redesign
2023-02-28 12:57:27 -05:00
Lai Jiang
85308eb975
Ignore invalid old CRL when performing update. (#1946)
There is no point comparing the old CRL to the new ones when the old one
is invalid. This could happen when the CA cert rotates, after which the
old CRL stop being valid as it fails signature verification against the
new cert.

This change will allow us to keep updating the CRL after a CA rotation without
having to manually delete the old CRL from the database.

See b/270983553.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1946)
<!-- Reviewable:end -->
2023-02-28 10:00:18 -05:00
Lai Jiang
ed62f27a4a
Update kythe vnames mapping (#1944) 2023-02-27 17:09:57 -05:00
Ben McIlwain
75851399ba
Remove "letter G with stroke" from Extended Latin IDN table (#1938)
ICANN doesn't like this character because it's confusable with a normal G (the
stroke tends to get lost in the visual clutter of the descender), and .com's
Extended Latin table doesn't use it either. Best to get rid of it.
2023-02-23 16:27:15 -05:00
Lai Jiang
6d54c8d113
Add allowed license for json (#1942)
For some reason `./gradlew clean build` on master is failing for me on
multiple machines due to a new org.json:json version triggering license
violations, even though the lock files are not changing.

Note that the old versions are still present because if I remove
"The JSON license", which the old versions use, the check also fails...
2023-02-23 11:37:31 -05:00
Lai Jiang
34dfa2760e
Add a table to record EPP resources needing DNS refresh (#1940) 2023-02-22 14:18:28 -05:00
Lai Jiang
ff39a4a763
Change default beam job region (#1937)
For reasons that I cannot explain, the same expand recurring billing
event pipeline would fail in us-east1 but succeed in us-central1.

See:

https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-02-09_14_52_24-162498476138221714;graphView=0?project=domain-registry

https://pantheon.corp.google.com/dataflow/jobs/us-east1/2023-02-09_14_26_07-4564782062878841960;graphView=1?project=domain-registry

Also improved how the accuracy of the metrics:

It is observed that both counters are consistently higher for the same
start and end times when running in dry run mode. There is no way to
test for consistency when not running in dry run, for obviously reasons.

I can make the recurrings in scope counter consistent by not updating it
in a side-effect-causing transaction, but there is no way around the
other counter. It can only be trusted when running in dry run mode,
unfortunately.
2023-02-13 15:57:32 -05:00
gbrodman
b1cd8c5a6f
Add a frontend endpoint for retrieving a domain in JSON form (#1916)
We might (likely will) modify some of the fiddly bits around this (maybe
the GSON serialization, where we do the actual authorization, etc) but
this should be a decent basic shell structure for endpoints that the new
registrar console can call to retrieve JSON results.
2023-02-09 15:09:42 -05:00
gbrodman
28c7bc3085
Generate and use an IAP-enabled ID token in the proxy (#1926)
This is only generated and used if "iapClientId" is set in the proxy
config. If so, we use code similar to
https://cloud.google.com/iap/docs/authentication-howto#obtaining_an_oidc_token_for_the_default_service_account
to generate an ID token that is valid for IAP. We set the token on the
Proxy-Authorization header so that we can keep using the pre-existing
access token as well -- IAP allows for us to use either the
Authorization header or the Proxy-Authorization header.
2023-02-09 14:50:35 -05:00
gbrodman
f36d22f4b1
Allow null GAIA IDs for User objects (#1933)
We were under the mistaken impression before that there was a reliable
way to, out-of-band, get a GAIA ID for a particular email address.
Unfortunately, that isn't the case (at least, not in a scalable way or
one that support agents could use). As a result, we have to allow null
GAIA IDs in the database.

When we (or the support team) create new users, we will only specify the
email address and not the GAIA ID. Then, when the user logs in for the
first time, we will have the GAIA ID from the provided ID token, and we
can populate it then.
2023-02-08 16:10:34 -05:00
Lai Jiang
ef3ce79b8a
Install procps in schema-deployer image (#1934)
It turns out this one uses pgrep and pkill as well, go figure...

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1934)
<!-- Reviewable:end -->
2023-02-08 09:59:47 -05:00
Lai Jiang
85317e3982
Update TMCH root certificate (#1918)
See b/260945047.

Also refactored the corresponding tests, which should future updates easier.

This change should be deployed at or around 2023-02-15T16:00:00Z.
2023-02-06 22:39:54 -05:00
Lai Jiang
a53b71ecd5
Install procps (#1932)
The schema verifier script needs pgrep and pkill, which do not come with
Debian.
2023-02-06 19:45:04 -05:00
Lai Jiang
fc9446876f
Install curl (#1931)
Tested by running "docker build .".
2023-02-06 16:45:52 -05:00
dependabot[bot]
654b165dff
Bump http-cache-semantics from 4.1.0 to 4.1.1 in /console-webapp (#1929)
Bumps [http-cache-semantics](https://github.com/kornelski/http-cache-semantics) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/kornelski/http-cache-semantics/releases)
- [Commits](https://github.com/kornelski/http-cache-semantics/compare/v4.1.0...v4.1.1)

---
updated-dependencies:
- dependency-name: http-cache-semantics
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 13:22:50 -05:00
Lai Jiang
14d68d4cb2
Change base image for schema-verifier and schema-deployer (#1930)
Ubuntu 18.04 is entering EOL and the Cloud Build jobs are failing,
seemingly due to connection error to 18.04 repos:

https://pantheon.corp.google.com/cloud-build/builds;region=global/126a7c90-4322-41f1-ba1c-a10e38a32dab;step=5?project=domain-registry-dev

We use Debian 10 for the main builder, so it's better to keep everything
on the same schedule:

https://cs.opensource.google/nomulus/nomulus/+/master:release/builder/Dockerfile

Debian 10 is supported till June 2024:

https://wiki.debian.org/LTS
2023-02-06 13:09:37 -05:00
Lai Jiang
bbf405d566
Fix expand recurring billing event pipeline (#1928) 2023-02-06 11:33:57 -05:00
sarahcaseybot
356f7d0099
Modify DomainCreateFlow to check for an applicable defaultPromoToken (#1904)
* Modify DomainCreateFlow to check for an applicable defaultPromoToken

* Add handling for deleted tokens

* Change cache to allocation token cache

* Abstract away cache methods

* Use AllocationToken.getAll in create flow

* Filter out empty tokens
2023-02-01 14:53:51 -05:00
dependabot[bot]
70509cfe46
Bump ua-parser-js from 0.7.31 to 0.7.33 in /console-webapp (#1924)
Bumps [ua-parser-js](https://github.com/faisalman/ua-parser-js) from 0.7.31 to 0.7.33.
- [Release notes](https://github.com/faisalman/ua-parser-js/releases)
- [Changelog](https://github.com/faisalman/ua-parser-js/blob/master/changelog.md)
- [Commits](https://github.com/faisalman/ua-parser-js/compare/0.7.31...0.7.33)

---
updated-dependencies:
- dependency-name: ua-parser-js
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lai Jiang <jianglai@google.com>
2023-01-31 14:52:26 -05:00
sarahcaseybot
5e081f4692
Prevent ending package tokens with active domains (#1919)
* Prevent ending package tokens with active domains

* Fix bad formatting in comments

* Fix lots of nits
2023-01-30 16:13:23 -05:00
Lai Jiang
07b87bbb4d
Remove @IdAllocation annotation from repoId (#1923)
This annotation only works for Long or long field.
2023-01-30 15:40:40 -05:00
gbrodman
6fabbb62d2
Use the Proxy-Authorization header when using nomulus + IAP (#1921) 2023-01-26 15:16:32 -05:00
Lai Jiang
d8a882daa0
Add fields needed to implement pull queue alternative (#1915) 2023-01-25 15:26:00 -05:00
Pavlo Tkach
de8c6fd316
Add a condition update precaution to validateNewState (#1920) 2023-01-25 14:53:12 -05:00
Weimin Yu
ae68917bdd
Upgrade to Gradle 7.3.2 (#1922)
This is an 'easy' upgrade that requires a minor change in
common/build.gradle and the removal of an unnecessary import in buildSrc.

Gradle 7.4 and above has breaking changes that break the latest nebula lint plugin. We may have to wait a while.
2023-01-25 12:47:35 -05:00
Lai Jiang
0736137a22
Update ExpandRecurringBillingEventsAction to use the beam pipeline (#1907)
Due to the way the beam pipeline is designed, it will expand an
recurring billing event when its event time is in scope for expansion,
instead of billing time. This means that the one time will be generated
45 days earlier. This would negate the need to check if the expansion is
finished when generating monthly invoices.

We will need to backfill the past 45 days of onetimes before the new
code is deployed. As an illustration, with the old code, a cursor time
of 2023-01-17 means that all auto-renewals whose billing time is before
2023-01-17 were created, which corresponds to an effective cursor time
of 2022-12-03 (45 days before 2023-01-17) for event time. This cursor
will need to be brought to 2023-01-17 to ensure that there is no gap in
generated event times when switching to use the new code.
2023-01-23 19:08:04 -05:00
Pavlo Tkach
c4b7929506
Remove not null constraint request_log_id column (#1917) 2023-01-23 09:37:20 -05:00