Support subscripting of arbitrary types, not only arrays.

This patch generalizes the subscripting infrastructure so that any
data type can be subscripted, if it provides a handler function to
define what that means.  Traditional variable-length (varlena) arrays
all use array_subscript_handler(), while the existing fixed-length
types that support subscripting use raw_array_subscript_handler().
It's expected that other types that want to use subscripting notation
will define their own handlers.  (This patch provides no such new
features, though; it only lays the foundation for them.)

To do this, move the parser's semantic processing of subscripts
(including coercion to whatever data type is required) into a
method callback supplied by the handler.  On the execution side,
replace the ExecEvalSubscriptingRef* layer of functions with direct
calls to callback-supplied execution routines.  (Thus, essentially
no new run-time overhead should be caused by this patch.  Indeed,
there is room to remove some overhead by supplying specialized
execution routines.  This patch does a little bit in that line,
but more could be done.)

Additional work is required here and there to remove formerly
hard-wired assumptions about the result type, collation, etc
of a SubscriptingRef expression node; and to remove assumptions
that the subscript values must be integers.

One useful side-effect of this is that we now have a less squishy
mechanism for identifying whether a data type is a "true" array:
instead of wiring in weird rules about typlen, we can look to see
if pg_type.typsubscript == F_ARRAY_SUBSCRIPT_HANDLER.  For this
to be bulletproof, we have to forbid user-defined types from using
that handler directly; but there seems no good reason for them to
do so.

This patch also removes assumptions that the number of subscripts
is limited to MAXDIM (6), or indeed has any hard-wired limit.
That limit still applies to types handled by array_subscript_handler
or raw_array_subscript_handler, but to discourage other dependencies
on this constant, I've moved it from c.h to utils/array.h.

Dmitry Dolgov, reviewed at various times by Tom Lane, Arthur Zakirov,
Peter Eisentraut, Pavel Stehule

Discussion: https://postgr.es/m/CA+q6zcVDuGBv=M0FqBYX8DPebS3F_0KQ6OVFobGJPM507_SZ_w@mail.gmail.com
Discussion: https://postgr.es/m/CA+q6zcVovR+XY4mfk-7oNk-rF91gH0PebnNfuUjuuDsyHjOcVA@mail.gmail.com

3 files changed, 32 insertions, 8 deletions

diff --git a/src/test/regress/sql/arrays.sql b/src/test/regress/sql/arrays.sql
index 891491e0466..c40619a8d5d 100644
--- a/src/test/regress/sql/arrays.sql
+++ b/src/test/regress/sql/arrays.sql
@@ -109,7 +109,7 @@ select ('[0:2][0:2]={{1,2,3},{4,5,6},{7,8,9}}'::int[])[1:2][2];
 --
 -- check subscription corner cases
 --
--- More subscripts than MAXDIMS(6)
+-- More subscripts than MAXDIM (6)
 SELECT ('{}'::int[])[1][2][3][4][5][6][7];
 -- NULL index yields NULL when selecting
 SELECT ('{{{1},{2},{3}},{{4},{5},{6}}}'::int[])[1][NULL][1];
diff --git a/src/test/regress/sql/opr_sanity.sql b/src/test/regress/sql/opr_sanity.sql
index 307aab1deb7..4189a5a4e09 100644
--- a/src/test/regress/sql/opr_sanity.sql
+++ b/src/test/regress/sql/opr_sanity.sql
@@ -34,7 +34,8 @@ begin
   if $2 = 'pg_catalog.any'::pg_catalog.regtype then return true; end if;
   if $2 = 'pg_catalog.anyarray'::pg_catalog.regtype then
     if EXISTS(select 1 from pg_catalog.pg_type where
-              oid = $1 and typelem != 0 and typlen = -1)
+              oid = $1 and typelem != 0 and
+              typsubscript = 'pg_catalog.array_subscript_handler'::pg_catalog.regproc)
     then return true; end if;
   end if;
   if $2 = 'pg_catalog.anyrange'::pg_catalog.regtype then
@@ -59,7 +60,8 @@ begin
   if $2 = 'pg_catalog.any'::pg_catalog.regtype then return true; end if;
   if $2 = 'pg_catalog.anyarray'::pg_catalog.regtype then
     if EXISTS(select 1 from pg_catalog.pg_type where
-              oid = $1 and typelem != 0 and typlen = -1)
+              oid = $1 and typelem != 0 and
+              typsubscript = 'pg_catalog.array_subscript_handler'::pg_catalog.regproc)
     then return true; end if;
   end if;
   if $2 = 'pg_catalog.anyrange'::pg_catalog.regtype then
diff --git a/src/test/regress/sql/type_sanity.sql b/src/test/regress/sql/type_sanity.sql
index 5e433388cdc..8c6e614f20a 100644
--- a/src/test/regress/sql/type_sanity.sql
+++ b/src/test/regress/sql/type_sanity.sql
@@ -63,12 +63,13 @@ WHERE p1.typtype not in ('p') AND p1.typname NOT LIKE E'\\_%'
            p2.typelem = p1.oid and p1.typarray = p2.oid)
 ORDER BY p1.oid;
 
--- Make sure typarray points to a varlena array type of our own base
+-- Make sure typarray points to a "true" array type of our own base
 SELECT p1.oid, p1.typname as basetype, p2.typname as arraytype,
-       p2.typelem, p2.typlen
+       p2.typsubscript
 FROM   pg_type p1 LEFT JOIN pg_type p2 ON (p1.typarray = p2.oid)
 WHERE  p1.typarray <> 0 AND
-       (p2.oid IS NULL OR p2.typelem <> p1.oid OR p2.typlen <> -1);
+       (p2.oid IS NULL OR
+        p2.typsubscript <> 'array_subscript_handler'::regproc);
 
 -- Look for range types that do not have a pg_range entry
 SELECT p1.oid, p1.typname
@@ -323,6 +324,26 @@ WHERE p1.typarray = p2.oid AND
     p2.typalign != (CASE WHEN p1.typalign = 'd' THEN 'd'::"char"
                          ELSE 'i'::"char" END);
 
+-- Check for typelem set without a handler
+
+SELECT p1.oid, p1.typname, p1.typelem
+FROM pg_type AS p1
+WHERE p1.typelem != 0 AND p1.typsubscript = 0;
+
+-- Check for misuse of standard subscript handlers
+
+SELECT p1.oid, p1.typname,
+       p1.typelem, p1.typlen, p1.typbyval
+FROM pg_type AS p1
+WHERE p1.typsubscript = 'array_subscript_handler'::regproc AND NOT
+    (p1.typelem != 0 AND p1.typlen = -1 AND NOT p1.typbyval);
+
+SELECT p1.oid, p1.typname,
+       p1.typelem, p1.typlen, p1.typbyval
+FROM pg_type AS p1
+WHERE p1.typsubscript = 'raw_array_subscript_handler'::regproc AND NOT
+    (p1.typelem != 0 AND p1.typlen > 0 AND NOT p1.typbyval);
+
 -- Check for bogus typanalyze routines
 
 SELECT p1.oid, p1.typname, p2.oid, p2.proname
@@ -356,7 +377,7 @@ SELECT t.oid, t.typname, t.typanalyze
 FROM pg_type t
 WHERE t.typbasetype = 0 AND
     (t.typanalyze = 'array_typanalyze'::regproc) !=
-    (typelem != 0 AND typlen < 0)
+    (t.typsubscript = 'array_subscript_handler'::regproc)
 ORDER BY 1;
 
 -- **************** pg_class ****************
@@ -452,7 +473,8 @@ WHERE o.opcmethod != 403 OR
     ((o.opcintype != p1.rngsubtype) AND NOT
      (o.opcintype = 'pg_catalog.anyarray'::regtype AND
       EXISTS(select 1 from pg_catalog.pg_type where
-             oid = p1.rngsubtype and typelem != 0 and typlen = -1)));
+             oid = p1.rngsubtype and typelem != 0 and
+             typsubscript = 'array_subscript_handler'::regproc)));
 
 -- canonical function, if any, had better match the range type