1# Signature Formats 2 3This document describes the signature file format created and used by metalava, 4doclava, apicheck, etc. 5 6There are currently 3 versions of this format: 7 81. The format emitted by doclava, and used for Android's signature files up 9 through Android P. Note that this isn't actually a single format; it evolved 10 over time, so older signature files vary a bit (many of these changes were 11 due to bugs getting fixed, such as type parameters missing from classes 12 and methods until they start appearing), and some were deliberate changes, 13 such as dropping the "final" modifier in front of every member if the 14 containing class is final. 15 162. The "new" format, which is described below, and is used in Android Q. This 17 format adds new information, such as annotations, parameter names and default 18 values, as well as cleans up a number of things (such as dropping 19 java.lang. prefixes on types, etc) 20 213. This is format v2, but will all nullness annotations replaced by a 22 Kotlin-syntax, e.g. "?" for nullable types, "!" for unknown/platform types, 23 and no suffix for non-nullable types. The initial plan was to include this 24 in format v2, but it was deferred since type-use annotations introduces 25 somple complexities in the implementation. 26 27 28## Motivation 29 30Why did we change from the historical doclava signature format (v1) 31to a new format? 32 33In order to support Kotlin better (though this will also benefit Java 34developers), we'd like to have nullness annotations (as well as some other 35annotations) be a formal part of the SDK. 36 37That means the annotations should be part of the signature files too -- such 38that we can not just record explicitly what the API contract is, but also 39enforce that changes are not only deliberate changes but also compatible 40changes. (For example, you can change the return value of a final method from 41nullable to non null, but not the other way around.) 42 43And if we were going to change the signature format, we might as well make some 44other changes too. 45 46 47### Comments 48 49In v2, line comments (starting with //) are allowed. This allows us to leave 50reminders and other issues with the signature source (though the update-api task 51will generally blow these away, so use sparingly.) 52 53### Header 54 55New signature files (v2+) generally include a file header comment which states 56the version number. This makes it possible for tools to more safely interpret 57signature files. For example, in v3 the type "String" means "@NonNull String", 58but in v2 "String" means "String with unknown nullness". 59 60The header looks like this: 61 62``` 63// Signature format: 2.0 64``` 65 66Here "2" is the major format version; the .0 allows for compatible minor 67variations of the format. 68 69### Include Annotations 70 71The new signature format now includes annotations; not all annotations (such as 72@Override etc); only those which are significant for the API, such as nullness 73annotations, etc. 74 75Annotations are included on the same line as the class/field/method, right 76before the modifiers. 77 78Here's how this looks: 79 80 81``` 82 method @Nullable public static Integer compute1(@Nullable java.util.List<java.lang.String>); 83``` 84 85 86(Notice how the annotations are not using fully qualified name; that's discussed 87below.) 88 89The annotations to be included are annotations for annotation types that are not 90hidden, and have class file or runtime retention. 91 92The annotations should be sorted alphabetically by fully qualified name. 93 94 95### Use Special Syntax or Nullness Annotations 96 97(Note: Only in version format 3+) 98 99As a special optimization, since we eventually want **all** APIs to have 100explicit nullness, use Kotlin's syntax for nullness. That means that for 101nullable elements, we add "?" after the type, for unknown nullness we add "!", 102and otherwise there's no suffix. In other words: 103 104 105<table> 106 <tr> 107 <td> 108 </td> 109 <td>Java Type 110 </td> 111 <td>Signature File Type 112 </td> 113 </tr> 114 <tr> 115 <td>Nullable 116 </td> 117 <td>@Nullable String 118 </td> 119 <td>String? 120 </td> 121 </tr> 122 <tr> 123 <td>Not nullable 124 </td> 125 <td>@NonNull String 126 </td> 127 <td>String 128 </td> 129 </tr> 130 <tr> 131 <td>Unknown nullability 132 </td> 133 <td>String 134 </td> 135 <td>String! 136 </td> 137 </tr> 138</table> 139 140 141The above signature line is turned into 142 143 144``` 145 method public Integer? compute1(java.util.List<java.lang.String!>?); 146``` 147 148 149### Clean Up Terminology 150 151Format v2 also cleans up some of the terminology used to describe the class 152structure in the signature file. For example, in v1, an interface is called an 153"abstract interface"; an interface extending another interface is said to 154"implement" it instead of "extend"-ing it, etc; enums and annotations are just 155referred to as classes that extend java.lang.Enum, or java.lang.Annotation etc. 156 157With these changes, these lines from v1 signature files: 158 159 160``` 161 public abstract interface List<E> implements java.util.Collection { ... } 162 public class TimeUnit extends java.lang.Enum { ... } 163 public abstract class SuppressLint implements java.lang.annotation.Annotation { ... } 164``` 165 166 167are replaced by 168 169 170``` 171 public interface List<E> extends java.util.Collection<E> { ... } 172 public enum TimeUnit { ... } 173 public @interface SuppressLint { ... } 174``` 175 176 177 178### Use Generics Everywhere 179 180The v1 signature files uses raw types in some places but not others. Note that 181in the above it was missing from super interface Collection: 182 183 184``` 185 public abstract interface List<E> implements java.util.Collection { ... } 186``` 187 188 189 whereas in the v2 format it's included: 190 191 192``` 193 public interface List<E> extends java.util.Collection<E> { ... } 194``` 195 196 197Similarly, v1 used erasure in throws clauses. For example, for this method: 198 199 200``` 201 public <X extends Throwable> T orElseThrow(Supplier<? extends X> exceptionSupplier) throws X 202``` 203 204v1 used this signature: 205 206 207``` 208 method public <X extends java.lang.Throwable> T orElseThrow(java.util.function.Supplier<? extends X>) throws java.lang.Throwable; 209``` 210 211Note how that's "throws Throwable" instead of "throws X". This results in b/110302703. 212 213In the v2 format we instead use the correct throws type: 214 215``` 216 method public <X extends java.lang.Throwable> T orElseThrow(java.util.function.Supplier<? extends X>) throws X; 217``` 218 219 220### Support Annotations 221 222The old format was completely missing annotation type methods: 223 224``` 225 public static abstract class ViewDebug.ExportedProperty implements java.lang.annotation.Annotation { 226 } 227``` 228 229We need to include annotation member methods, as well as their default values 230since those are API-significant. Here's how this looks in the v2 file format 231(also applying the @interface terminology change described above) : 232 233 234``` 235 public static @interface ViewDebug.ExportedProperty { 236 method public abstract String category() default ""; 237 method public abstract boolean deepExport() default false; 238 method public abstract android.view.ViewDebug.FlagToString[] flagMapping() default {}; 239 method public abstract boolean formatToHexString() default false; 240 method public abstract boolean hasAdjacentMapping() default false; 241 method public abstract android.view.ViewDebug.IntToString[] indexMapping() default {}; 242 method public abstract android.view.ViewDebug.IntToString[] mapping() default {}; 243 method public abstract String prefix() default ""; 244 method public abstract boolean resolveId() default false; 245 } 246``` 247 248 249### Support Kotlin Modifiers 250 251This doesn't currently apply to the SDK, but the signature files are also used 252in the support library, and some of these are written in Kotlin and exposes 253Kotlin-specific APIs. 254 255That means the v2 format can express API-significant aspects of Kotlin. This 256includes special modifiers, such as sealed, inline, operator, infix, etc: 257 258``` 259 method public static operator int get(android.graphics.Bitmap, int x, int y); 260 method public static infix android.graphics.Rect and(android.graphics.Rect, android.graphics.Rect r); 261``` 262 263### Support Kotlin Properties 264 265Kotlin's Java support means that it wil take a Kotlin property and compile it 266into getters and setters which you can call from Java. But you cannot calls 267these getters and setters from Kotlin; you **must** use the property 268syntax. Therefore, we need to also capture properties in the signature files. If 269you have this Kotlin code: 270 271 272``` 273 var property2: String? = "initial" 274``` 275 276it will get recorded in the signature files like this: 277 278``` 279 property public java.lang.String? property2 = "initial"; 280 method public java.lang.String? getProperty2(); 281 method public void setProperty2(java.lang.String? p); 282``` 283 284The last two elements are "redundant"; they could be computed from the property 285name (and included if the property declaration uses special annotations to name 286the getters and setters away from the defaults), but it's helpful to be explicit 287(and this allows us to specify the default value). 288 289### Support Named Parameters 290 291Kotlin supports default values for parameters, and these are a part of the API 292contract, so we need to include them in the signature format. 293 294Here's an example: 295 296``` 297 method public static void edit(android.content.SharedPreferences, boolean commit); 298``` 299 300In v1 files we only list type names, but in v2 we allow an optional parameter 301name to be specified; "commit" in the above. 302 303Note that this isn't just for Kotlin. Just like there are special nullness 304annotations to mark up the null contract for an element, we will also have a 305special annotation to explicitly name a Java parameter: 306@android.annotation.ParameterName (which is hidden). This obviously isn't usable 307from Java, but Kotlin client code can now reference the parameter. 308 309Therefore, the following Java code (not signature code) will also produce 310exactly the same signature as the above: 311 312``` 313 public static void edit(SharedPreferences prefs, @ParameterName("commit") boolean ct) {…} 314``` 315 316(Note how the implementation parameter doesn't have to match the public, API 317name of the parameter.) 318 319### Support Default Values 320 321In addition to named parameters, Kotlin also supports default values. These are 322also be part of the v2 signature since (as an example) removing a default value 323is a compile-incompatible change. 324 325Therefore, the v2 format allows default values to be specified after the type 326and/or parameter name: 327 328``` 329 method public static void edit(SharedPreferences, boolean commit = false); 330``` 331 332For Kotlin code, the default parameter values are extracted automatically, and 333for Java, just as with parameter names, you can specify a special annotation to 334record the default value for usage from languages that support default parameter 335values: 336 337``` 338 public static void edit(SharedPreferences prefs, @DefaultValue("false") boolean ct) {…} 339``` 340 341 342### Include Inherited Methods 343 344Consider a scenario where a public class extends a hidden class, and that hidden 345class defines a public method. 346 347Doclava did not include these methods in the signature files, but they **were** 348present in the stub files (and therefore part of the API). In the v2 signature 349file format, we include these. 350 351An example of this is StringBuilder#setLength. According to the old signature 352files, that method does not exist, but clearly it's there in the SDK. The reason 353this happens is that StringBuilder is a public class which extends hidden class 354AbstractStringBuilder, which defines the public method setLength. 355 356 357### No Hardcoded Enum Methods 358 359Doclava always inserted two special methods in the signature files for every 360enum: values() and valueOf(): 361 362``` 363 public static final class CursorJoiner.Result extends java.lang.Enum { 364 method public static android.database.CursorJoiner.Result valueOf(java.lang.String); 365 method public static final android.database.CursorJoiner.Result[] values(); 366 enum_constant public static final android.database.CursorJoiner.Result BOTH; 367 enum_constant public static final android.database.CursorJoiner.Result LEFT; 368 enum_constant public static final android.database.CursorJoiner.Result RIGHT; 369 } 370``` 371 372It didn't do that in stubs, because you can't: those are special methods 373generated by the compiler. There's no reason to list these in the signature 374files since they're entirely implied by the enum, you can't change them, and 375it's just extra noise. 376 377In the new v2 format these are no longer present: 378 379``` 380 public static enum CursorJoiner.Result { 381 enum_constant public static final android.database.CursorJoiner.Result BOTH; 382 enum_constant public static final android.database.CursorJoiner.Result LEFT; 383 enum_constant public static final android.database.CursorJoiner.Result RIGHT; 384 } 385``` 386 387### Remove "deprecated" Modifier 388 389The old signature file format used "deprecated" as if it was a modifier. In the 390new format, we instead list these using annotations, @Deprecated. 391 392### Standard Modifier Order 393 394Doclava had a "random" (but stable) order of modifiers. 395 396In the new signature format, we're using the standard modifier order for Java 397and Kotlin, wihch more closely mirrors what is done in the source code. 398 399Version format 1 order: 400 401``` 402public/protected/private default static final abstract synchronized transient volatile 403``` 404 405Version format 2 order: 406 407``` 408public/protected/internal/private abstract default static final transient volatile synchronized 409``` 410 411The above list doesn't include the Kotlin modifiers, which are inserted 412according to the Kotlin language style guide: 413https://kotlinlang.org/docs/reference/coding-conventions.html#modifiers 414 415### Sort Classes By Fully Qualified Names 416 417In "extends" lists, the signature file can list a comma separated list of 418classes. The classes are listed by fully qualified name, but in v1 it was sorted 419by simple name. In the v2 format, we sort by fully qualified name instead. 420 421### Use Wildcards Consistently 422 423Doclava (v1) would sometimes use the type bound <?> and other times use <? 424extends Object>. These are equivalent. In the v2 format, <? extends Object> is 425always written as <?>. 426 427### Annotation Simple Names 428 429We have a number of annotations which are significant for the API -- not just 430the nullness as deprecation ones (which are specially supported in v3 via the 431?/! Kotlin syntax and the deprecated "modifier"), but annotations for permission 432requirements, range constraints, valid constant values for an integer, and so 433on. 434 435In the codebase, these are typically in the android.annotation. package, 436referencing annotation classes that are generally **not** part of the API. When 437we generate the SDK, we translate these into publicly known annotations, 438androidx.annotation, such that Studio, lint, the Kotlin compiler and others can 439recognize the metadata. 440 441That begs the question: which fully qualified name should we put in the 442signature file? The one that appeared in the source (which is hidden, or in the 443case of Kotlin code, a special JetBrains nullness annotation), or the one that 444it gets translated into? 445 446In v2 we do neither: We use only the simple name of the annotations in the 447signature file, for annotations that are in the well known packages. In other 448words, instead of any of these alternative declarations: 449 450``` 451 method public void setTitleTextColor(@android.annotation.ColorInt int); 452 method public void setTitleTextColor(@android.support.annotation.ColorInt int); 453 method public void setTitleTextColor(@androidx.annotation.ColorInt int); 454``` 455 456in v2 we have simply 457 458``` 459 method public void setTitleTextColor(@ColorInt int); 460``` 461 462### Simple Names in Java.lang 463 464In Java files, you can implicitly reference classes in java.lang without 465importing them. In v2 offer the same thing in signature files. There are several 466classes from java.lang that are used in lots of places in the signature file 467(java.lang.String alone is present in over 11,000 lines of the API file), and 468other common occurrences are java.lang.Class, java.lang.Integer, 469java.lang.Runtime, etc. 470 471This basically builds on the same idea from having an implicit package for 472annotations, and doing the same thing for java.lang: Omitting it when writing 473signature files, and implicitly adding it back when reading in signature files. 474 475This only applies to the java.lang package, not any subpackages, so for example 476java.lang.reflect.Method will **not** be shortened to reflect.Method. 477 478### Type Use Annotations 479 480In v3, "type use annotations" are supported which means annotations can appear 481within types. 482 483### Skipping some signatures 484 485If a method overrides another method, and the signatures are the same, the 486overriding method is left out of the signature file. This basically compares the 487modifiers, ignoring some that are not API significant (such as "native"). Note 488also that some modifiers are implicit; for example, if a method is implementing 489a method from an interface, the interface method is implicitly abstract, so the 490implementation will be included in the signature file. 491 492In v2, we take this one step further: If a method differs **only** from its 493overridden method by "final", **and** if the containing class is final, then the 494method is not included in the signature file. The same is the case for 495deprecated. 496 497### Miscellaneous 498 499Some other minor tweaks in v2: 500 501* Fix formatting for package private elements. These had two spaces of 502 indentation; this is probably just a bug. The new format aligns their 503 indentation with all other elements. 504* Don't add spaces in type bounds lists (e.g. Map<X,Y>, not Map<X, Y>.) 505 506## Historical API Files 507 508Metalava can read and write these formats. To switch output formats, invoke it 509with for example --format=v2. 510 511The Android source tree also has checked in versions of the signatures for all 512the previous API levels. Metalava can regenerate these for a new format. 513For example, to update all the signature files to v3, run this command: 514 515``` 516$ metalava --write-android-jar-signatures *<android source dir>* --format=v3 517``` 518