google-nomulus/docs/coding-faq.md

Coding FAQ

This file is a motley assortment of informational emails generated in response to questions from development partners.

Which entities are backed up using commit logs?

Short answer: Check the file backup_kinds.txt.

Long answer: There are really two axes. Our custom Ofy actually doesn't condition commit logging on the @Entity class being mutated. Rather, it conditions based on:

1. the methods you're invoking
2. annotations on the class being mutated

Methods: The standard save() and delete() methods always write commit logs. The alternate saveWithoutBackup() and deleteWithoutBackup() methods never write commit logs (and have appropriately scary names to avoid accidental use). This makes it clear what you're getting at the callsite.

Annotations: There are regular @Entity-annotated entities, @VirtualEntity-annotated entities, and @NotBackedUp-annotated entities. An @VirtualEntity is a "virtual entity" that just serves to construct parent keys for other entities (e.g. EppResourceIndexBucket) and is never written to datastore itself. An @NotBackedUp-annotated entity is one that specifically shouldn't be backed up (like the commit log entities themselves).

We don't actually prevent you from not-backing-up a regular entity, because sometimes that's necessary (e.g. in the restore logic, and some other miscellaneous places). We do prevent you from trying to save/delete an @VirtualEntity (with or without backups) or save/delete an @NotBackedUp entity with backups (i.e. using the regular save()/delete() calls), since these are always usage errors.

We went with an @NotBackedUp annotation versus an @BackedUp annotation largely because the not-backed-up case is more of the special case (as explained above, the annotation itself doesn't directly turn backups on or off). To keep track of what we're intending to back up, we:

1. synthesize a list of all the backed-up kinds by filtering the registered entity classes to remove those with @VirtualEntity and @NotBackedUp annotations: ExportConstants,
2. check this list into the repo: backup_kinds.txt, then
3. run an enforcement test to make sure it is up to date: ExportConstantsTest.

How do I mock Google Cloud Storage in tests?

AppEngine's GCS client automatically switches over to a local implementation, GcsServiceFactory, for tests.

So rather than mocking GCS-related stuff at all, just use the fake local implementation. This is what our tests should be doing; see ExportCommitLogDiffActionTest.

Very rarely there have been cases where we've needed something beyond that (e.g. to test against GCS being eventually consistent). In that case, rather than mocking GcsUtils, you'd need to create a real instance of it but pass in a mocked-out GcsService instance. Those are a bit of a pain to make since GcsServiceImpl itself is also final (and not code we control), but you could roll your own implementation by hand, or cheat and use a reflective proxy, as we do in GcsDiffFileListerTest.

How do I test authentication on the SDK Development Server?

Can someone explain how GaeUserIdConverter.convertEmailAddressToGaeUserId() actually does the conversion? I see it's doing a save/load(/delete) of a GaeUserIdConverter, which contains a User. Does Objectify do some magic on load to look up the real GAE user ID from the email address? In trying to get the registry to run in the SDK Development Server, I am seeing the wrong user ID when adding a new RegistryContact using the command line tool.

The App Engine development server is not particularly robust; it appears that it always returns userId 185804764220139124118 for any authenticated user, as per this StackOverflow thread.

For testing purposes, it might suffice to just create a RegistrarContact with that userId by hand somehow, so that you can log in. In the longer term, if we switch to Google Sign-In, this specific problem would go away, but based on the above, it looks like OAuthService doesn't really work on the dev server either.

So for testing actual "real" authentication, you'd want to use an alpha instance rather than the development server. We don't use the development server very much internally for this reason.

Do you support RDAP?

We are working on an implementation of the Registry Data Access Protocol (RDAP), ICANN's proposed successor to WHOIS, which provides similar data to WHOIS, but in a structured format. The standard is defined in RFCs 7480 through 7484:

• RFC 7480: HTTP Usage in the Registration Data Access Protocol (RDAP)
• RFC 7481: Security Services for the Registration Data Access Protocol (RDAP)
• RFC 7482: Registration Data Access Protocol (RDAP) Query Format
• RFC 7483: JSON Responses for the Registration Data Access Protocol (RDAP)
• RFC 7484: Finding the Authoritative Registration Data (RDAP) Service

Some gaps in the implementation remain, but most of the functionality is currently available. If you access this endpoint on a running Nomulus system:

https://{PROJECT-ID}.appspot.com/rdap/domains?name=ex*

it should search for all domains that start with "ex", returning the results in JSON format. This functionality is still under development, so it is quite possible that the format of returned data will change over time, but the basic structure should be the same, as defined by RFCs 7480 through 7484. Request paths which ought to mostly work (though no guarantees yet):

/rdap/domain/abc.tld
/rdap/nameserver/ns1.abc.tld
/rdap/entity/ROID
/rdap/entity/registrar-iana-identifier
/rdap/domains?name=abc.tld
/rdap/domains?name=abc*
/rdap/domains?name=abc*.tld
/rdap/domains?nsLdhName=ns1.abc.tld
/rdap/domains?nsLdhName=ns*
/rdap/domains?nsIp=1.2.3.4
/rdap/nameservers?name=ns*.abc.tld
/rdap/nameservers?ip=1.2.3.4
/rdap/entities?fn=John*
/rdap/entities?handle=ROI*
/rdap/entities?handle=registrar-iana-identifier

The wildcard searches allow only trailing wildcards, with the exception that you can specify a TLD after the domain name wildcard (e.g. abc*.tld), and you can specify .domain.tld after the nameserver wildcard (e.g. ns*.domain.tld). But you can't do anything else, like searching for nameservers with ns*.tld. When using a wildcard, we currently require a prefix of at least two characters, to avoid having someone search for *. There are other limitations to the system which we plan to address in the future.

How do I embed a new class in an existing Datastore entity class?

In Objectify 4.x, which we are pinned to, you have to annotate a class with @Embed in order to natively nest it within an entity, e.g. as a field within an @Entity class. This behavior is different from how things work in Objectify 5.x, which is what the current documentation reflects. Unfortunately the 4.x docs are hard to access, but you can search this old copy for "embed" to see the details:

https://raw.githubusercontent.com/objectify/objectify-legacy-wiki/v4/Entities.wiki

So you'd have to annotate your subobject with @Embed for it to work. It's possible in Objectify 4.x to have a class that's both @Embed and @Entity, but it'd be a little unorthodox. Usually it'd be cleaner to either make it purely @Embed (so purely a container for data within the parent entity) or purely @Entity, in which case you wouldn't use it as a field of the other @Entity but would instead refer to it by key, e.g. you'd store a Key instead. For objects which don't change very often, it is preferable to use the @Entity approach.

How do I resolve Bazel package visibility problems?

As a short-term solution, you can disable the bazel visibility checking with --nocheck_visibility. We are investigating ways to clarify the way that custom functionality is expected to be added, which will involve changing the package visibility accordingly.