Extend ALTER OPERATOR to allow setting more optimization attributes.

Allow the COMMUTATOR, NEGATOR, MERGES, and HASHES attributes to be set
by ALTER OPERATOR.  However, we don't allow COMMUTATOR/NEGATOR to be
changed once set, nor allow the MERGES/HASHES flags to be unset once
set.  Changes like that might invalidate plans already made, and
dealing with the consequences seems like more trouble than it's worth.
The main use-case we foresee for this is to allow addition of missed
properties in extension update scripts, such as extending an existing
operator to support hashing.  So only transitions from not-set to set
states seem very useful.

This patch also causes us to reject some incorrect cases that formerly
resulted in inconsistent catalog state, such as trying to set the
commutator of an operator to be some other operator that already has a
(different) commutator.

While at it, move the InvokeObjectPostCreateHook call for CREATE
OPERATOR to not occur until after we've fixed up commutator or negator
links as needed.  The previous ordering could only be justified by
thinking of the OperatorUpd call as a kind of ALTER OPERATOR step;
but we don't call InvokeObjectPostAlterHook therein.  It seems better
to let the hook see the final state of the operator object.

In the documentation, move the discussion of how to establish
commutator pairs from xoper.sgml to the CREATE OPERATOR ref page.

Tommy Pavlicek, reviewed and editorialized a bit by me

Discussion: https://postgr.es/m/CAEhP-W-vGVzf4udhR5M8Bdv88UYnPrhoSkj3ieR3QNrsGQoqdg@mail.gmail.com

3 files changed, 144 insertions, 52 deletions

diff --git a/doc/src/sgml/ref/alter_operator.sgml b/doc/src/sgml/ref/alter_operator.sgml
index a4a1af564ff..673dcce2f50 100644
--- a/doc/src/sgml/ref/alter_operator.sgml
+++ b/doc/src/sgml/ref/alter_operator.sgml
@@ -30,7 +30,11 @@ ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</repla
 ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</replaceable> | NONE } , <replaceable>right_type</replaceable> )
     SET ( {  RESTRICT = { <replaceable class="parameter">res_proc</replaceable> | NONE }
            | JOIN = { <replaceable class="parameter">join_proc</replaceable> | NONE }
-         } [, ... ] )
+           | COMMUTATOR = <replaceable class="parameter">com_op</replaceable>
+           | NEGATOR = <replaceable class="parameter">neg_op</replaceable>
+           | HASHES
+           | MERGES
+          } [, ... ] )
 </synopsis>
  </refsynopsisdiv>
 
@@ -121,23 +125,95 @@ ALTER OPERATOR <replaceable>name</replaceable> ( { <replaceable>left_type</repla
      </listitem>
    </varlistentry>
 
+   <varlistentry>
+    <term><replaceable class="parameter">com_op</replaceable></term>
+    <listitem>
+     <para>
+      The commutator of this operator. Can only be changed if the operator
+      does not have an existing commutator.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><replaceable class="parameter">neg_op</replaceable></term>
+    <listitem>
+     <para>
+      The negator of this operator. Can only be changed if the operator does
+      not have an existing negator.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>HASHES</literal></term>
+    <listitem>
+     <para>
+      Indicates this operator can support a hash join. Can only be enabled and
+      not disabled.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>MERGES</literal></term>
+    <listitem>
+     <para>
+      Indicates this operator can support a merge join. Can only be enabled
+      and not disabled.
+     </para>
+    </listitem>
+   </varlistentry>
+
   </variablelist>
  </refsect1>
 
  <refsect1>
+  <title>Notes</title>
+
+  <para>
+   Refer to <xref linkend="xoper"/> and
+   <xref linkend="xoper-optimization"/> for further information.
+  </para>
+
+  <para>
+   Since commutators come in pairs that are commutators of each other,
+   <literal>ALTER OPERATOR SET COMMUTATOR</literal> will also set the
+   commutator of the <replaceable class="parameter">com_op</replaceable>
+   to be the target operator.  Likewise, <literal>ALTER OPERATOR SET
+   NEGATOR</literal> will also set the negator of
+   the <replaceable class="parameter">neg_op</replaceable> to be the
+   target operator.  Therefore, you must own the commutator or negator
+   operator as well as the target operator.
+  </para>
+ </refsect1>
+
+ <refsect1>
   <title>Examples</title>
 
   <para>
    Change the owner of a custom operator <literal>a @@ b</literal> for type <type>text</type>:
 <programlisting>
 ALTER OPERATOR @@ (text, text) OWNER TO joe;
-</programlisting></para>
+</programlisting>
+  </para>
 
   <para>
-    Change the restriction and join selectivity estimator functions of a custom operator <literal>a &amp;&amp; b</literal> for type <type>int[]</type>:
+   Change the restriction and join selectivity estimator functions of a
+   custom operator <literal>a &amp;&amp; b</literal> for
+   type <type>int[]</type>:
 <programlisting>
-ALTER OPERATOR &amp;&amp; (_int4, _int4) SET (RESTRICT = _int_contsel, JOIN = _int_contjoinsel);
-</programlisting></para>
+ALTER OPERATOR &amp;&amp; (int[], int[]) SET (RESTRICT = _int_contsel, JOIN = _int_contjoinsel);
+</programlisting>
+  </para>
+
+  <para>
+   Mark the <literal>&amp;&amp;</literal> operator as being its own
+   commutator:
+<programlisting>
+ALTER OPERATOR &amp;&amp; (int[], int[]) SET (COMMUTATOR = &amp;&amp;);
+</programlisting>
+  </para>
 
  </refsect1>
 
diff --git a/doc/src/sgml/ref/create_operator.sgml b/doc/src/sgml/ref/create_operator.sgml
index e27512ff391..c421fd21e9d 100644
--- a/doc/src/sgml/ref/create_operator.sgml
+++ b/doc/src/sgml/ref/create_operator.sgml
@@ -104,7 +104,7 @@ CREATE OPERATOR <replaceable>name</replaceable> (
   </para>
 
   <para>
-   The other clauses specify optional operator optimization clauses.
+   The other clauses specify optional operator optimization attributes.
    Their meaning is detailed in <xref linkend="xoper-optimization"/>.
   </para>
 
@@ -112,7 +112,7 @@ CREATE OPERATOR <replaceable>name</replaceable> (
    To be able to create an operator, you must have <literal>USAGE</literal>
    privilege on the argument types and the return type, as well
    as <literal>EXECUTE</literal> privilege on the underlying function.  If a
-   commutator or negator operator is specified, you must own these operators.
+   commutator or negator operator is specified, you must own those operators.
   </para>
  </refsect1>
 
@@ -231,7 +231,67 @@ COMMUTATOR = OPERATOR(myschema.===) ,
   <title>Notes</title>
 
   <para>
-   Refer to <xref linkend="xoper"/> for further information.
+   Refer to <xref linkend="xoper"/> and <xref linkend="xoper-optimization"/>
+   for further information.
+  </para>
+
+  <para>
+   When you are defining a self-commutative operator, you just do it.
+   When you are defining a pair of commutative operators, things are
+   a little trickier: how can the first one to be defined refer to the
+   other one, which you haven't defined yet?  There are three solutions
+   to this problem:
+
+   <itemizedlist>
+    <listitem>
+     <para>
+      One way is to omit the <literal>COMMUTATOR</literal> clause in the
+      first operator that you define, and then provide one in the second
+      operator's definition.  Since <productname>PostgreSQL</productname>
+      knows that commutative operators come in pairs, when it sees the
+      second definition it will automatically go back and fill in the
+      missing <literal>COMMUTATOR</literal> clause in the first
+      definition.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      Another, more straightforward way is just to
+      include <literal>COMMUTATOR</literal> clauses in both definitions.
+      When <productname>PostgreSQL</productname> processes the first
+      definition and realizes that <literal>COMMUTATOR</literal> refers to
+      a nonexistent operator, the system will make a dummy entry for that
+      operator in the system catalog.  This dummy entry will have valid
+      data only for the operator name, left and right operand types, and
+      owner, since that's all that <productname>PostgreSQL</productname>
+      can deduce at this point.  The first operator's catalog entry will
+      link to this dummy entry.  Later, when you define the second
+      operator, the system updates the dummy entry with the additional
+      information from the second definition.  If you try to use the dummy
+      operator before it's been filled in, you'll just get an error
+      message.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      Alternatively, both operators can be defined
+      without <literal>COMMUTATOR</literal> clauses
+      and then <command>ALTER OPERATOR</command> can be used to set their
+      commutator links.  It's sufficient to <command>ALTER</command>
+      either one of the pair.
+     </para>
+    </listitem>
+   </itemizedlist>
+
+   In all three cases, you must own both operators in order to mark
+   them as commutators.
+  </para>
+
+  <para>
+   Pairs of negator operators can be defined using the same methods
+   as for commutator pairs.
   </para>
 
   <para>
diff --git a/doc/src/sgml/xoper.sgml b/doc/src/sgml/xoper.sgml
index a929ced07d7..954a90d77d0 100644
--- a/doc/src/sgml/xoper.sgml
+++ b/doc/src/sgml/xoper.sgml
@@ -146,44 +146,6 @@ SELECT (a + b) AS c FROM test_complex;
      <literal>=</literal> operator must specify that it is valid, by marking the
      operator with commutator information.
     </para>
-
-    <para>
-     When you are defining a self-commutative operator, you just do it.
-     When you are defining a pair of commutative operators, things are
-     a little trickier: how can the first one to be defined refer to the
-     other one, which you haven't defined yet?  There are two solutions
-     to this problem:
-
-     <itemizedlist>
-      <listitem>
-       <para>
-        One way is to omit the <literal>COMMUTATOR</literal> clause in the first operator that
-        you define, and then provide one in the second operator's definition.
-        Since <productname>PostgreSQL</productname> knows that commutative
-        operators come in pairs, when it sees the second definition it will
-        automatically go back and fill in the missing <literal>COMMUTATOR</literal> clause in
-        the first definition.
-       </para>
-      </listitem>
-
-      <listitem>
-       <para>
-        The other, more straightforward way is just to include <literal>COMMUTATOR</literal> clauses
-        in both definitions.  When <productname>PostgreSQL</productname> processes
-        the first definition and realizes that <literal>COMMUTATOR</literal> refers to a nonexistent
-        operator, the system will make a dummy entry for that operator in the
-        system catalog.  This dummy entry will have valid data only
-        for the operator name, left and right operand types, and result type,
-        since that's all that <productname>PostgreSQL</productname> can deduce
-        at this point.  The first operator's catalog entry will link to this
-        dummy entry.  Later, when you define the second operator, the system
-        updates the dummy entry with the additional information from the second
-        definition.  If you try to use the dummy operator before it's been filled
-        in, you'll just get an error message.
-       </para>
-      </listitem>
-     </itemizedlist>
-    </para>
    </sect2>
 
    <sect2 id="xoper-negator">
@@ -217,12 +179,6 @@ SELECT (a + b) AS c FROM test_complex;
     <literal>x &lt;&gt; y</literal>.  This comes up more often than you might think, because
     <literal>NOT</literal> operations can be inserted as a consequence of other rearrangements.
    </para>
-
-   <para>
-    Pairs of negator operators can be defined using the same methods
-    explained above for commutator pairs.
-   </para>
-
   </sect2>
 
   <sect2 id="xoper-restrict">