Rfc for implicit inclusion of checked header files. (checkedc#998)

* Final version of the rfc for implicit inclusion of checked header files.

* Modified the solution to handle the case where a system header file may have clang-specific declarations.

* Update to the approach to handle the presence of clang-specific declarations.

* Changes in the checkedc-clang repo for supporting implicit inclusion of checked headers.

* Added __checkedc flag, added test cases and addressed review comments.

* Certain header files like threads.h, unistd.h, arpa/inet.h and sys/socket.h
may not be found in all compilation environments (Ex. Windows).
Therefore the includes in the wrapper header files need to be
guarded by __has_include_next. This may impact the expected output of a
testcase.

* Incorporated review comments. Also added another test case.

* Temporarily marking five of 3C tests as unsupported on Windows as they
are crashing, in order to test all the other test cases on the ADO build
machines.

* isFunctionProtoType should be used with getAs, not dyn_cast.

* Enabling the 3C test cases.

Co-authored-by: Matt McCutchen (Correct Computation) <[email protected]>

Author: sulekhark

clang/docs/checkedc/rfc/Include-Checked-Hdrs.md

@@ -0,0 +1,182 @@
+# Explicit and Implicit Inclusion of Checked Header Files
+Author: Sulekha Kulkarni, Approver: David Tarditi, Date Approved:
+
+
+# OARP
+
+| Role          | Person |
+|-------------------|---------------|
+| Owner         | Sulekha Kulkarni  |
+| Approver      | David Tarditi     |
+| Reviewers     | Joseph Lloyd, Mandeep Grang, Katherine Kjeer |
+| Reviewers     | Mike Hicks and other folks at CCI |
+
+
+## Goal
+The goal of this document is to record the design considerations for allowing
+programs to include the checked counterparts of system header files either
+explicitly or implicitly during the conversion of these programs to Checked C.
+
+## Problem Summary
+The current approach for including checked header files, which is to explicitly
+replace an included header file with its checked counterpart, is inconvenient
+for adoption to Checked C - all the system header files included in a project
+need to be altered. So we need a way to implicitly include checked header
+files.
+
+## Conditions that a Solution must Satisfy
+ - The solution must provide users an option to either opt into the implicit
+ inclusion of checked header files, or opt out of it if checked header files
+ are implicitly included by default. This flexibility is required to support
+ custom build environments.
+ - The solution must be easy to integrate into an existing build system.
+ - The current approach of explicitly including checked header files must
+ always remain a viable alternative.
+ - The order in which the compiler driver searches the include paths must remain
+ unaltered.
+ - The solution must be able to accommodate any clang-specific declarations in
+ system header files, if present (Ex.: inttypes.h).
+ - The Checked-C-specific declarations must live in the checkedc repository.
+
+
+## Details of the Current Approach 
+ - The current approach supports only the explicit inclusion of checked header
+ files.
+ - An installed Checked C compiler places the checked counterparts of system
+ header files like `stdio_checked.h` in the directory
+ `$INSTALL_DIR/lib/clang/<VERSION>/include`, which clang searches before it
+ looks in system include directories. Note that there is no alteration of the
+ include search path to accommodate any include files related to Checked C.
+ - If a checked header file like `stdio_checked.h` is included in a program,
+ clang picks it up from the above location.
+
+## Solution Space
+ - Solution 1:
+     - Let `foo.h` be a system header file with a checked counterpart called
+     `foo_checked.h`. We add a new file, also called `foo.h`, in the same
+     directory that contains `foo_checked.h`. This new `foo.h` should contain
+     `#include_next <foo.h>` plus all the Checked-C-specific declarations
+     present in `foo_checked.h`. In addition, the original contents of
+     `foo_checked.h` should be deleted and it should now only contain
+     `#include <foo.h>`.
+     - The way this solution will work:
+          - If a program includes `foo.h`, clang will first pick up the
+          "checked" version of `foo.h`. As this `foo.h` has a
+          `#include_next <foo.h>`, the system `foo.h` will be picked up
+          (infinite recursion is avoided because of include_next), and all the
+          Checked-C-specific declarations in the "checked" version of `foo.h`
+          will also be picked up.
+          - If the program includes `foo_checked.h`, the line `#include <foo.h>`
+          in `foo_checked.h` will initiate the same process as above.
+     - **Pros**: 1) The current approach of explicit inclusion is supported.
+       2) No changes are required to the build system. 3) No changes are
+       required to the order in which the include directories are searched.
+       4) It is convenient for automation.
+     - **Cons**: 1) The solution does not provide a way to opt out of including
+       checked headers. 2) While it is easy to accommodate clang-specific
+       declarations in system header files (Checked-C-specific declarations can
+       be added after the clang-specific declarations), this solution will
+       require some Checked-C-specific declarations to be part of checkedc-clang
+       repository.
+
+ - Solution 2:
+     - Have a different compiler driver to provide the facility for implicit
+       inclusion of header files.
+     - **Pros**: 1) The solution provides the option to opt into implicitly
+       including checked header files. 2) The existing approach of explicit
+       inclusion is supported. 3) In most cases integration into a build system
+       will be easy: the `CC` variable can be redefined appropriately. 4) It is
+       convenient for automation.
+     - **Cons**: 1) It is a very heavy-weight solution. 2) In some cases
+       integrating with a build system will be more involved when both the
+       drivers are needed to compile the sources. This may happen in a scenario
+       where most source files want implicit inclusion but a few source files
+       want explicit inclusion because they want to directly include some system
+       header files in order to avoid Checked-C-specific declarations.
+
+ - Solution 3:
+     - Let `foo.h` be a system header file with its checked counterpart called
+       `foo_checked.h`. We add a new file called `foo.h` in the same directory
+       that contains `foo_checked.h`. In this new file, we add the following:
+
+             #ifdef IMPLICIT_INCLUDE_CHECKED_HDRS
+             #include <foo_checked.h>
+             #else
+             #include_next <foo.h>
+             #endif
+     - In `foo_checked.h`, we modify `#include <foo.h>` to
+       `#include_next <foo.h>`.
+
+     - The way this solution will work:
+        - If a program includes `foo.h`, clang will pick up either
+          `foo_checked.h` or the system `foo.h` depending on whether or not
+          `-DIMPLICIT_INCLUDE_CHECKED_HDRS` is specified on the compilation
+          commandline.  Therefore specifying this flag will cause the implicit
+          inclusion of the checked counterpart of `foo.h`, which will make it
+          convenient for automation. Not specifying the commandline flag will
+          not perform implicit inclusion, providing a mechanism to opt out of 
+          implicit inclusion. Infinite recursion is avoided because of
+	  `#include_next`.
+        - If a program includes `foo_checked.h` current behavior will prevail.
+          The above flag has no effect on the explicit inclusion of checked
+          header files.
+
+     - **Pros**: 1) The solution provides a way to opt into implicit inclusion
+       of checked header files. 2) The existing approach of explicit inclusion
+       is supported. 3) In most cases integration with the build system is easy:
+       the flag `-DIMPLICIT_INCLUDE_CHECKED_HDRS` is passed to the compilation
+       of all source files through the `CFLAGS` variable. 4) No changes are
+       required to the order in which include directories are searched. 5) It
+       is convenient for automation.
+
+     - **Cons**: 1) In some cases integration with the build system will be more
+       involved if the above flag needs to be passed to the compilation of most
+       source files and avoided for a few source files that may want to directly
+       include system header files in order to avoid Checked-C-specific
+       declarations. 2) It is difficult to accomodate clang-specific
+       declarations.
+
+## Proposed Solution:
+
+The proposed solution is as follows:
+  - We will implicitly include checked header files by default and provide the 
+    compilation flag NO_IMPLICIT_INCLUDE_CHECKED_HDRS to opt out of the implicit
+    inclusion.
+  - For header files that do not have clang-specific declarations, but have
+    Checked-C-specific declarations (there are 13 such header files at present),
+    we will do the following:
+      - For each system header file, say `foo.h`, that does not have
+        clang-specific declarations but has Checked-C-specific declarations, we
+        will add a new file also called `foo.h` that will contain the following:
+
+            #if !defined __checkedc || defined NO_IMPLICIT_INCLUDE_CHECKED_HDRS
+            #include_next <foo.h>
+            #else
+            #include <foo_checked.h>
+            #endif
+
+      - The file `foo_checked.h` will contain `#include_next <foo.h>` and the
+        Checked-C-specific declarations.
+
+  - For a system header file that contains clang-specific declarations and also
+    Checked-C-specific declarations (there is one such header file at present),
+    we will do the following:
+      - Let `bar.h` be such a header file. Then there already 
+        exists a file called `bar.h` in the `clang/lib/Headers` directory,
+        which contains clang-specific declarations in the following format:
+
+            #include_next <bar.h>
+            <currently existing clang-specific declarations>
+
+        At the end of this pre-existing `bar.h`, we will add the following:
+
+            #if defined __checkedc && !defined NO_IMPLICIT_INCLUDE_CHECKED_HDRS
+            #include <bar_checked_internal.h>
+            #endif
+
+      - All the Checked-C-specific declarations (that are currently present in
+        `bar_checked.h`) will be moved to `bar_checked_internal.h`. The file
+        `bar_checked.h` will be modified to contain just the following:
+
+            #include <bar.h>
+            #include <bar_checked_internal.h>

clang/lib/3C/ConstraintVariables.cpp

@@ -234,7 +234,7 @@ PointerVariableConstraint::PointerVariableConstraint(
           OrigType = FD->getType().getTypePtr();
         }
         if (OrigType->isFunctionProtoType()) {
-          const FunctionProtoType *FPT = dyn_cast<FunctionProtoType>(OrigType);
+          const FunctionProtoType *FPT = OrigType->getAs<FunctionProtoType>();
           AnalyzeITypeExpr = (FPT->getReturnType() == QT);
         }
       }

clang/lib/3C/Utils.cpp

@@ -195,7 +195,7 @@ bool filePathStartsWith(const std::string &Path, const std::string &Prefix) {
 bool functionHasVarArgs(clang::FunctionDecl *FD) {
   if (FD && FD->getFunctionType()->isFunctionProtoType()) {
     const FunctionProtoType *SrcType =
-        dyn_cast<FunctionProtoType>(FD->getFunctionType());
+        FD->getFunctionType()->getAs<FunctionProtoType>();
     return SrcType->isVariadic();
   }
   return false;

clang/lib/Frontend/InitPreprocessor.cpp

@@ -374,6 +374,8 @@ static void InitializeStandardPredefinedMacros(const TargetInfo &TI,
       Builder.defineMacro("__STDC_VERSION__", "199901L");
     else if (!LangOpts.GNUMode && LangOpts.Digraphs)
       Builder.defineMacro("__STDC_VERSION__", "199409L");
+    if (LangOpts.CheckedC)
+      Builder.defineMacro("__checkedc", "202104L");
   } else {
     //   -- __cplusplus
     //      [C++20] The integer literal 202002L.

clang/lib/Headers/inttypes.h

@@ -18,6 +18,11 @@
 #error MSVC does not have inttypes.h prior to Visual Studio 2013
 #endif
 
+#ifdef __checkedc
+#pragma CHECKED_SCOPE push
+#pragma CHECKED_SCOPE off
+#endif
+
 #include_next <inttypes.h>
 
 #if defined(_MSC_VER) && _MSC_VER < 1900
@@ -94,4 +99,14 @@
 #define SCNxFAST32 "x"
 #endif
 
+#ifdef __checkedc
+#pragma CHECKED_SCOPE pop
+#endif
+
+#if defined __checkedc && !defined NO_IMPLICIT_INCLUDE_CHECKED_HDRS
+// If compiling for Checked C and if implicit inclusion of checked headers is
+// enabled.
+#include <inttypes_checked_internal.h>
+#endif
+
 #endif /* __CLANG_INTTYPES_H */

clang/test/3C/basic.c

@@ -58,7 +58,8 @@ char *basic2(int temp) {
     return 0;
   }
 }
-//CHECK: char *basic2(int temp) : itype(_Ptr<char>) {
+//CHECK_ALL: char *basic2(int temp) : itype(_Nt_array_ptr<char>) {
+//CHECK_NOALL: char *basic2(int temp) : itype(_Ptr<char>) {
 //CHECK_ALL: char data _Nt_checked[17] =  "abcdefghijklmnop";
 //CHECK_ALL: char data2 _Nt_checked[65] =
 //CHECK: char *buffer = malloc<char>(8);
@@ -95,7 +96,8 @@ void sum_numbers(int count) {
   }
   free(ptr);
 }
-//CHECK: int *ptr = (int *)malloc<int>(n * sizeof(int));
+//CHECK_NOALL: int *ptr = (int *)malloc<int>(n * sizeof(int));
+//CHECK_ALL: _Array_ptr<int> ptr : count(n) = (_Array_ptr<int>)malloc<int>(n * sizeof(int));
 
 void basic_calloc(int count) {
   int n, i, sum = 0;
@@ -119,7 +121,8 @@ void basic_calloc(int count) {
   printf("Sum = %d", sum);
   free(ptr);
 }
-//CHECK: int *ptr = (int *)calloc<int>(n, sizeof(int));
+//CHECK_NOALL: int *ptr = (int *)calloc<int>(n, sizeof(int));
+//CHECK_ALL: _Array_ptr<int> ptr : count(n) = (_Array_ptr<int>)calloc<int>(n, sizeof(int));
 
 void basic_realloc(int count) {
   int i, n1, n2;
@@ -144,7 +147,8 @@ void basic_realloc(int count) {
 
   free(ptr);
 }
-//CHECK: int *ptr = (int *)malloc<int>(n1 * sizeof(int));
+//CHECK_NOALL: int *ptr = (int *)malloc<int>(n1 * sizeof(int));
+//CHECK_ALL: _Array_ptr<int> ptr : count(n1) = (_Array_ptr<int>)malloc<int>(n1 * sizeof(int));
 
 struct student {
   char name[30];

clang/test/3C/linkedlist.c

@@ -26,14 +26,14 @@ void display(List *list);
 void reverse(List *list);
 //CHECK: void reverse(_Ptr<List> list);
 void destroy(List *list);
-//CHECK: void destroy(List *list : itype(_Ptr<List>));
+//CHECK: void destroy(_Ptr<List> list);
 
 struct node {
 
   int data;
 
   struct node *next;
-  //CHECK: struct node *next;
+  //CHECK: _Ptr<struct node> next;
 };
 
 struct list {
@@ -48,7 +48,7 @@ Node *createnode(int data) {
   //CHECK: _Ptr<Node> createnode(int data) {
 
   Node *newNode = malloc(sizeof(Node));
-  //CHECK: _Ptr<Node> newNode = malloc<Node>(sizeof(Node));
+  //CHECK: _Ptr<Node> newNode =  malloc<Node>(sizeof(Node));
 
   if (!newNode) {
 
@@ -169,7 +169,7 @@ void reverse(List *list) {
 }
 
 void destroy(List *list) {
-  //CHECK: void destroy(List *list : itype(_Ptr<List>)) {
+  //CHECK: void destroy(_Ptr<List> list) { 
 
   Node *current = list->head;
 

clang/test/CheckedC/headers/nostdinc.c

@@ -0,0 +1,53 @@
+// The -nostdinc option suppresses the search for include files in standard
+// system include directories like /usr/local/include, /usr/include,
+// /usr/include/<TARGET_ARCH>, and also the <BUILD>/lib/clang/<VERSION>/include
+// directory.
+// Therefore, in the context of a Checked C compilation, none of the
+// header files listed below will be found when -nostdinc is specified on the
+// compilation command line.
+//
+// This test confirms the above behavior.
+//
+// RUN: %clang -target x86_64-unknown-unknown \
+// RUN:   -nostdinc -ffreestanding -fsyntax-only %s
+
+#if defined(__has_include)
+
+#if __has_include (<assert.h>) \
+ || __has_include (<errno.h>) \
+ || __has_include (<fenv.h>) \
+ || __has_include (<inttypes.h>) \
+ || __has_include (<math.h>) \
+ || __has_include (<signal.h>) \
+ || __has_include (<stdio.h>) \
+ || __has_include (<stdlib.h>) \
+ || __has_include (<string.h>) \
+ || __has_include (<threads.h>) \
+ || __has_include (<time.h>) \
+ || __has_include (<checkedc_extensions.h>) \
+ || __has_include (<unistd.h>) \
+ || __has_include (<sys/socket.h>) \
+ || __has_include (<arpa/inet.h>)
+#error "expected to *not* be able to find standard C headers"
+#endif
+
+
+#if __has_include (<assert_checked.h>) \
+ || __has_include (<errno_checked.h>) \
+ || __has_include (<fenv_checked.h>) \
+ || __has_include (<inttypes_checked.h>) \
+ || __has_include (<math_checked.h>) \
+ || __has_include (<signal_checked.h>) \
+ || __has_include (<stdio_checked.h>) \
+ || __has_include (<stdlib_checked.h>) \
+ || __has_include (<string_checked.h>) \
+ || __has_include (<threads_checked.h>) \
+ || __has_include (<time_checked.h>) \
+ || __has_include (<checkedc_extensions.h>) \
+ || __has_include (<unistd_checked.h>) \
+ || __has_include (<sys/socket_checked.h>) \
+ || __has_include (<arpa/inet_checked.h>)
+#error "expected to *not* be able to find Checked C headers"
+#endif
+
+#endif