ISRAEL JOURNAL OF MATHEMATICS 126 (2001), 173–237

GENERIC REPRESENTATIONS FOR THE UNITARY GROUP IN THREE VARIABLES

BY

Stephen Gelbart^{∗}

The Nicki and J. Ira Harris Professorial Chair, Department of Mathematics

The Weizmann Institute of Science, Rehovot 76100, Israel e-mail: gelbar@wisdom.weizmann.ac.il

AND

Hervé Jacquet†

Department of Mathematics, Columbia University New York, NY 10027, USA e-mail: hj@math.columbia.edu

AND

Jonathan Rogawski‡

Department of Mathematics, UCLA Los Angeles, CA 90024, USA e-mail: jonr@math.ucla.edu

ABSTRACT

We show that for the quasi-split unitary group in three variables every tempered packet of cuspidal automorphic representations contains a globally generic representation.

* Partially supported by a Minerva grant. † Partially supported by NSF grant DMS 9619766 and 9988611. ‡ Partially supported by NSF grant DMS 9700950.

Received August 2, 2000

Contents

1 Introduction 174 2 Complements on the Asai L-function 182 3 Local relative Bessel distributions 187

3.1 Proof: First step . . . . . . . . . . . . . . . . . . . 189 3.2 Proof: Second step . . . . . . . . . . . . . . . . . . 190 3.3 A result on local distinguished representations . . . . . . 192

4 Complements on orbital integrals 193 5 Estimates for functionals 196

5.1 Integral representation of a section . . . . . . . . . . . 196 5.2 Whittaker linear form . . . . . . . . . . . . . . . . . 199 5.3 Intertwining period integral . . . . . . . . . . . . . . 201 5.4 Intertwining operator . . . . . . . . . . . . . . . . . 205 5.5 Whittaker functional for GL(3) . . . . . . . . . . . . . 206

6 Limit formula for J_{χ}(f) 210 7 Contribution of the parabolic of type (2, 1) 214 8 Contribution of the Borel subgroup 216

8.1 The term J_{χ}^{min}(f) . . . . . . . . . . . . . . . . . . . 218 8.2 The term J_{χ}^{max}(f) . . . . . . . . . . . . . . . . . . 222

9 The discrete part of the trace formula 230 10 Convergence for the unitary group 231 11 Conclusion 233 References 234

1. Introduction Suppose that G is a reductive group, defined and quasi-split over a number field F. Let N be a maximal unipotent subgroup in G and θ a character of N(F_{𝔸}) trivial on N(F) and generic. Let π be a cuspidal automorphic representation of G(F_{𝔸}) and V the isotypic component of the space of cusp forms corresponding to π. If ϕ is a smooth vector in V we set

W(ϕ) := ∫_{N(F)\N(F_{𝔸})} ϕ(n)\overline{θ}(n)dn.

Thus W is a continuous linear form transforming under θ. We say that π is globally generic with respect to θ if the linear form W is not identically zero. If W is non-zero, then, for each place v, the component π_{v} is generic with respect to θ_{v}, that is, admits a non-zero linear form W_{v} transforming under θ_{v} (Whittaker linear form). Recall that such a form is unique, within a scalar factor. If we define W_{ϕ}(g) = ∫ ϕ(ng)\overline{θ}(n)dn then the space W spanned by the functions W_{ϕ} is irreducible. The closure V_{1} of the kernel of the map ϕ ↦ W_{ϕ} is invariant and so we may decompose V into an orthogonal direct sum V = V_{1}⊕V_{0}. The map ϕ ↦ W_{ϕ} is now injective on the space of smooth vectors of V_{0}. It follows that the representation on V_{0} is irreducible. It is generally conjectured that inside a tempered L-packet of automorphic representations, there is exactly one component which is generic with respect to a given generic character θ (cf.

[KRS] and [fS2]). The purpose of this paper is to establish the conjecture for the quasi-split group in three variables relative to a quadratic extension E/F. We note that in general we may have V ≠ V_{0}. However, here V = V_{0}, that is, the representation π has multiplicity one ([Ro1], Theorem 13.3.1, p. 201). Before explaining the proof, we also raise another, related, question. If π is as above, we can define a global Bessel distribution as follows:

J_{π}(f) := ∑_{ϕ} W(ρ(f)ϕ)\overline{W(ϕ)};

here the sum is over an orthonormal basis of V_{0}; it amounts to the same to take the sum over a basis of V, or even a basis of the space of the packet to which π belongs—if the conjecture is true. At a place v we may define a local Bessel distribution

B_{v}(f_{v}) := ∑_{ϕ} W_{v}(π(f_{v})ϕ)\overline{W_{v}(ϕ)}.

Note that the distribution is defined within a positive factor. Thus, to make the following statement precise, one would need to choose a normalization of this distribution. It follows from the local uniqueness that the global Bessel distribution decomposes as an infinite product of the local Bessel distributions:

(1)  J_{π}(f) = C ∏_{v} B_{v}(f_{v}), if f = ⊗f_{v}.

The constant C is positive as a ratio of positive type distributions. The question at hand is to compute the constant in terms of L-functions attached to π. In the case of the unitary group, it is clear that with more local information we could answer this question completely. See [BM] for another example.

The results of the present paper were announced in a short note with Solomon Friedberg as a coauthor ([FGJR]). It is a pleasure to acknowledge his contribution to the genesis of this work.

We now proceed to discuss the proof of the result. We let E/F be a quadratic extension of number fields with Galois group \{1, σ\}. We often write σ(z) = \overline{z}. We let U_{1} be the unitary group in one variable, that is, the group of elements of norm 1 in E^{×}. We denote by G the group GL(3) regarded as an algebraic group over E and we denote by Z its center. We let H be the group GL(3, F) regarded as an algebraic group over F and we denote by Z_{H} its center. To prove the result, we use the concept of a cuspidal distinguished representation of G(E_{𝔸}). In a precise way, we say that such a Π is distinguished by H(F_{𝔸}) (or simply distinguished), if the central character ω of Π is trivial on F_{𝔸}^{×} (that is, is distinguished), and there is a form ϕ in the space of Π such that

I(ϕ) := ∫_{Z_{H}(𝔸)H(F)\H(F_{𝔸})} ϕ(h)dh

is non-zero. On the one hand, it is a result of [F2], that, if Π is distinguished, then Π^{σ} = Π̃, where Π^{σ} is the representation defined by Π^{σ}(g) = Π(g^{σ}) and Π̃ denotes the representation contragredient to Π. On the other hand, it is a result of [F2] and [FZ], that Π is distinguished if and only if the (partial) Asai L-function attached to Π has a pole at s = 1. From this and the factorization of the Rankin–Selberg L-function attached to the pair (Π, Π^{σ}) in terms of Asai L-functions, it follows that if the central character of Π is distinguished and Π^{σ} = Π̃, then Π is distinguished (section 2). In turn, this condition of symmetry is equivalent to Π being the standard (or stable) base change of a tempered stable packet of automorphic representations of the unitary group. Thus for GL(3) at least, the property of Π being distinguished is equivalent to Π being the functorial image of a stable packet of automorphic cuspidal representations. This fact is predicated by the analysis of the potential pole of the Asai L-function in terms of the L-group ([F2]).

Our main tool is then the relative trace formula. The relative trace formula used here was first discussed in [Y] in the context of GL(2). In fact, the split unitary group in [Y] is replaced by the group GL(2, F). However, the functions at hand are in fact supported on the subgroup G′ of matrices with determinant a norm, and the group G′ is (up to central tori) isomorphic to the unitary group in two variables. In [F2], this is exploited to reformulate (and extend) the formula of [Y] in terms of the unitary group in two variables. Reference [F3] contains a study in the context of GL(n) of the formula used here in the context of GL(3). The motivation of [F3] is essentially the same as ours. Indeed, given the above discussion for GL(3), the conjecture stated in [F2] (which motivates [F3]) amounts to saying that every stable tempered packet of representations of the unitary group contains a generic representation. We use the discussion of [F3]. However, the analysis of the continuous spectrum presented in [F3], although suggestive, is insufficient and erroneous. We do provide the analysis for GL(3), which we hope is correct. It is based on the results of [JLR].

We now proceed to discuss the trace formula in question. Recall we denote by G the group GL(3) regarded as an algebraic group over E. We let B be the group of upper triangular matrices, A the group of diagonal matrices, N the group of upper triangular matrices with unit diagonal, and Z the group of scalar matrices in G. We write an element g as g = (g_{i,j}), where i is the column index (this is the opposite of the standard convention). We fix a non-trivial character ψ of F_{𝔸}/F, a character ζ of U_{1}(F_{𝔸})/U_{1}(F) and denote by ω the character of E_{𝔸}^{×}/E^{×} defined by ω(z) = ζ(z\overline{z}^{−1}).

If f is a smooth function of compact support on GL(3, E_{𝔸}) we define a kernel

(2)  K_{f}(x, y) = ∫_{Z(E_{𝔸})/Z(E)} ( ∑_{ξ∈GL(3,E)} f(x^{−1}ξyz))ω(z)dz.

Our main object of study is the distribution

(3)  J(f) := ∫_{H(F)Z_{H}(F_{𝔸})\H(F_{𝔸})} ∫_{N(E)\N(E_{𝔸})} K_{f}(h, n)θ(n)dndh.

Here θ is the character of N(E_{𝔸}) defined by

(4)  θ ( \begin{matrix}1 & x & z \\ 0 & 1 & y \\ 0 & 0 & 1 \end{matrix} ) = ψ_{E}(x − y), ψ_{E}(z) = ψ(z + \overline{z}).

The distribution can be computed in terms of the symmetric space

𝔖 = \{s ∈ GL(3, E): s\overline{s} = 1\}.

Indeed, we denote by P the polarization map

P: G → S, P(g) := \overline{g}^{−1}g.

The group G(E) operates on the right by twisted conjugation on 𝔖:

s ↦^{g} \overline{g}^{−1}sg.

There is a smooth function of compact support Φ on 𝔖(F_{𝔸}) such that

Φ(P(g)) = ∫_{H(F_{𝔸})} f(hg)dh.

Then

(5) J(f) = ∫_{N(E)\N(E_{𝔸})} ∫_{𝕌_{1}(F)\𝕌_{1}(F_{𝔸})} ( ∑_{ξ∈𝔖(F)} Φ(\overline{n}^{−1}ξnu))θ(n)dnζ(u)du.

In this paper, we will make the following simplifying assumption: we let S_{0} be a finite set of places of F and S be the corresponding set of places of E. We assume that S_{0} contains the places at infinity, the even finite places, the places which ramify in E, the places where ψ ramifies and the places where ζ ramifies. We assume that f is a product of local functions f_{v}. For v ∉ S we assume that f_{v} is bi-invariant under the standard maximal compact subgroup K_{v}. Furthermore, we assume that for almost all such v’s, the function f_{v} is the characteristic function of K_{v}. We make the further assumption that for all places v ∈ S_{0} which are inert in E and thus under one place v of E, the function f_{v} is supported on the open set Ω_{v} ⊂ GL(3, E_{v}) of matrices of the form hηan with h ∈ H_{v}, a ∈ A_{v}, n ∈ N_{v}. Here η is an element such that P(η) = w, and

(6)  w = ( \begin{matrix}0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \end{matrix} ) .

We do assume that there is at least one such place. The corresponding function Φ_{v}, defined by

Φ_{v}(P(g)) = ∫_{H_{v}} f_{v}(hg)dh,

is supported on the open set Ω_{v_{0}}^{w} := P(Ω_{v}). This is the set of matrices s ∈ 𝔖_{v_{0}} such that s_{1,3} ≠ 0. Every matrix s in Ω_{v_{0}}^{w} can be written uniquely in the form

s = \overline{n}^{−1}wud_{a}n

with n ∈ N(E_{v}), u a scalar matrix in U_{1}(F_{v_{0}}), and d_{a} = diag(a, 1, \overline{a}^{−1}). The complement of Ω_{v_{0}}^{w} is the union of orbits of N(E_{v}) with a non-trivial centralizer (cf. [F3]).

We consider the global orbital integrals, for u ∈ U_{1}(F_{𝔸}) and a ∈ E_{𝔸}^{×}

(7)  J(ud_{a}, Φ) := ∫_{N(E_{𝔸})} Φ(\overline{n}^{−1}wud_{a}n)θ(n)dn.

Then

J(f) = ∑_{α∈E^{×}} ∫_{U_{1}(F_{𝔸})} J(ud_{α}, Φ)ζ(u)du.

There is also a notion of local orbital integrals. If v_{0} is a place of F inert in E and v the corresponding place of E, then the orbital integral is

(8)  J(ud_{a}, Φ_{v_{0}}) := ∫_{N(E_{v})} Φ_{v_{0}}(\overline{n}^{−1}wud_{a}n)θ(n)dn.

If v_{0} splits into two places v_{1}, v_{2} of E, then 𝔖_{v_{0}} can be identified with the set of pairs (g_{1}, g_{2}) with g_{1}g_{2} = 1 and then

Φ_{v_{0}}(g_{2}^{−1}g_{1}, g_{1}^{−1}g_{2}) = ∫_{H_{v_{0}}} f_{v_{1}}(hg_{1})f_{v_{2}}(hg_{2})dh.

We choose one of the two places, v_{1} say. Then we may identify 𝔖_{v_{0}} with GL(3, F_{v_{0}}), via the map

(g_{1}, g_{2}) ↦ g_{1}.

We may also identify GL(3, E_{v_{1}}) and GL(3, E_{v_{2}}) with GL(3, F_{v_{0}}). Then

Φ_{v_{0}}(g_{1}) = ∫_{GL(3,F_{v_{0}} )} f_{v_{1}}(hg_{1})f_{v_{2}}(h)dh

so that Φ_{v_{0}} is the convolution of f_{v_{1}} and the function f̌_{v_{2}} defined by f̌_{v_{2}}(g) = f_{v_{2}}(g^{−1}). The orbital integral becomes

(9) J(zd_{a}, Φ_{v_{0}}) := ∫_{N(F_{v_{0}} )×N(F_{v_{0}} )} Φ_{v_{0}}(n_{2}^{−1}wzd_{a}n_{1})θ(n_{2})θ(n_{1})dn_{1}dn_{2},

where z is a central element, d_{a} = diag(a_{1}, 1, a_{2}^{−1}) and θ is defined by

θ ( \begin{matrix}1 & x & z \\ 0 & 1 & y \\ 0 & 0 & 1 \end{matrix} ) = ψ_{F_{v_{0}}} (x − y).

Likewise, we consider the unitary group U for the Hermitian matrix w. The group is defined by the equation^{t}\overline{g}wg = w. We denote by A′ the group of diagonal matrices, by N′ the group of upper triangular matrices with unit diagonal in U and by Z_{U} the center of U. Let f′ be a smooth function of compact support on U(F_{𝔸}). We construct a kernel

(10)  K′_{f′}(x, y) := ∫_{Z_{U} (F_{𝔸})/Z_{U} (F)} ∑_{ξ∈U(F)} f′(x^{−1}ξyz)ζ(z)dz,

and the distribution

(11)  J′(f′) := ∫ K_{f′}(n_{1}, n_{2})θ′(n_{1})θ′(n_{2})dn_{1}dn_{2}.

Here θ′ is the character of N′(F_{𝔸}) trivial on N′(F) defined by

θ′ ( \begin{matrix}1 & x & t + \frac{x\overline{x}}{2} \\ 0 & 1 & −\overline{x} \\ 0 & 0 & 1 \end{matrix} ) = ψ_{E}(x).

The group A′(F) acts by conjugation on N′ and thus on the set of characters N′(F_{𝔸}) trivial on N′(F). The group A′(F) is transitive on the set of generic characters. Thus the choice of θ′ is inessential. We will make the following assumptions. Let S_{0} and S be as above. Then we assume that for every place v_{0} ∈ S_{0} inert in E, the function f′_{v_{0}} is supported on the open set Ω′_{v_{0}} of matrices of the form n_{2}^{−1}wud_{a}n_{1}. For u ∈ U(F_{𝔸}) and a ∈ E_{𝔸}^{×} we consider the global orbital integral

(12)  J′(ud_{a}, f′) := ∫ f′(n_{1}^{−1}wud_{a}n_{2})θ(n_{1})θ(n_{2})dn_{1}dn_{2}.

Then

J′(f′) = ∑_{α} ∫ J′(ud_{α}, f′)ζ(u)du.

As before, we have a notion of local orbital integrals. At a place v_{0} inert in E,

(13) J′(ud_{a}, f′_{v_{0}}) := ∫_{N′(F_{v_{0}} )×N′(F_{v_{0}} )} f′_{v_{0}}(n_{1}^{−1}wud_{a}n_{2})θ′(n_{1})θ′(n_{2})dn_{1}dn_{2}.

If the place v_{0} splits into v_{1}, v_{2} then U_{v_{0}} is the group of pairs (g_{1}, g_{2}) with g_{2} = w^{t}g_{1}^{−1}w. Identifying as before U_{v_{0}} to GL(3, F_{v_{0}}) via the map (g_{1}, g_{2}) ↦ g_{1}, we see that the local orbital integral is again given by (9) with Φ_{v_{0}} replaced by f′_{v_{0}}.

In this paper we will consider pairs (f, f′) with matching orbital integrals, in the sense that

(14)  J(ud_{a}, Φ) = J′(ud_{a}, f′).

More precisely, we will assume that for every place v_{0} of F split in E we have

f′_{v_{0}} = Φ_{v_{0}},

where both sides are viewed as functions on GL(3, F_{v_{0}}). For v_{0} ∈ S_{0} inert in E, we shall assume that

(15)  J(ud_{a}, Φ_{v_{0}}) = J′(ud_{a}, f′_{v_{0}}).

In view of our simplifying assumption, given Φ_{v_{0}} there is f′_{v_{0}} such that (15) is true. Finally, we assume that for v_{0} ∉ S_{0} and inert, the function f′_{v_{0}} is the image of the function f_{v_{0}} by the base change homomorphism of the Hecke algebra. It is a theorem of [J2] and [Mao2] that relation (15) is then true. Thus relation (14) is true as well and we get

(16)  J(f) = J′(f′).

As usual, we decompose the kernel K and K′ with respect to cuspidal data. For cuspidal data χ for G (resp. χ′ for U) we obtain a kernel K_{χ} (resp. K′_{χ′}). We define a corresponding distribution J_{χ}(f) (resp. J_{χ′}(f′)). Let χ be a cuspidal automorphic representation of GL(3, E_{𝔸}) whose central character is ω. Then J_{χ}(f) = 0 unless χ is distinguished by H. Conversely, we verify in the third section that, if χ is distinguished by H, then there is f satisfying our simplifying assumptions such that J_{χ}(f) ≠ 0.

This being so, we consider a stable packet of automorphic representations of U with central character ζ. The base change of this packet is a single cuspidal representation χ of GL(3, E_{𝔸}) with central character ω where ω(z) = ζ(zz^{−σ}). Now χ is distinguished. It follows that we have an identity

(17)  J_{χ}(f) = ∑_{χ′} J_{χ′}(f),

where the sum on the right is over all χ′ such that the unramified representation ⨂_{v∉S} χ_{v} is the image of the unramified representation ⨂_{v_{0}∉S_{0}} χ′_{v_{0}} under the standard base change. For the group U, this amounts to saying that the sum is over all the members of the tempered stable packet whose image is χ. From the formula, it follows that there is at least one such χ′ such that J_{χ′} ≠ 0. This implies that χ′ is globally generic and proves the theorem.

A natural question is to ask whether the sum on the right of (17) has only one element. We can answer this question affirmatively in the last section, thanks to local results. Taking this result for granted, we have then an identity

(18)  J_{χ}(f) = J_{χ′}(f′).

In addition to the previous notations, we will also introduce the following subgroups and elements of GL(3):

(19) P := \{( \begin{matrix}∗ & ∗ & ∗ \\ ∗∗ & ∗ & ∗ \\ 0 & 0 & 1 \end{matrix} )\} , P_{1} = P Z,

(20) w_{1} = ( \begin{matrix}0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{matrix} ) , w_{2} = ( \begin{matrix}1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{matrix} ) .

The paper is arranged as follows. In section 2 we discuss a few needed complements on the Asai L-function. In section 3, we prove that the local relative Bessel distributions do not vanish on a suitable open set. In section 4, for the sake of completeness, we establish elementary results on orbital integrals, which are used here, and have also been used in previous references without proof. In section 5, we provide estimates for the functionals at hand. The analysis of the continuous spectrum of the trace formulas occupies sections 6 to 10. The main theorems are quickly established in section 11.

2. Complements on the Asai L-function

Let E/F be a quadratic extension of number fields with Galois group {1, σ}. Regard the group GL(n, E) as an algebraic group over F via restriction of scalars. The (Galois form of) the L-group of G is the semi-direct product:

LG := (GL(n, ℂ) × GL(n, ℂ)) × {1, σ}.

The Asai L-function is the Langlands L-function attached to the n^{2} dimensional twisted representation r: ^{L}G → GL(ℂ^{n} ⊗ ℂ^{n}) defined by

r(g_{1}, g_{2})(v ⊗ w) = (g_{1}v) ⊗ (g_{2}w),
r(σ)(v ⊗ w) = w ⊗ v.

Let Π be a cuspidal automorphic representation of GL(3). We fix S_{0} as before so that Π is unramified outside S. We denote by L^{S_{0}}(s, Π, Asai) the product of the local factors of the Asai L-function over the places not in S_{0}.

Lemma 1: Let μ be any idele class character of E such that μ|F_{𝔸}^{×} = ω_{E/F}. Then, if μ is unramified outside S,

L^{S}(s, Π ⊗ Π^{σ}) = L^{S_{0}}(s, Π, Asai)L^{S_{0}}(s, Π ⊗ μ, Asai).

Proof: The lemma is formal from the point of view of the L-group (cf. [HLR] and [Gol]). For the sake of completeness we provide a proof. We claim that the contributions of each place v_{0} of F not in S_{0} to the two sides are the same. If v_{0} is inert and Π_{v} is the unramified representation associated to n unramified quasi-characters χ_{i} of E, the contribution of v_{0} to the left hand side is

∏_{i} L(s, χ_{i}^{2}) ∏_{i<j} L(s, χ_{i}χ_{j})^{2},

while

L(s, Π_{v_{0}}, Asai) = ∏_{i} L(s, χ_{i}|F) ∏_{i<j} L(s, χ_{i}χ_{j}).

Since μ_{v}^{2} = 1, our assertion follows from the relation

L(s, χ_{i}^{2}) = L(s, χ_{i}|F)L(s, χ_{i}|F ω_{E/F}).

See also [Gol]. If v_{0} splits into two places v_{1}, v_{2} then q_{v_{0}} = q_{v_{1}} = q_{v_{2}}. If we write L(s, Π_{v_{i}}) = det [I − t(Π_{v_{i}})q_{v_{i}}^{−s}]^{−1} with t(Π_{v_{i}}) ∈ GL(n, ℂ), then the contribution of v_{0} to the left hand side is

det [I − t(Π_{v_{1}}) ⊗ t(Π_{v_{2}})q_{v_{0}}^{−s}]^{−2} .

On the other hand,

L(s, Π_{v_{0}}, Asai) = det [I − t(Π_{v_{1}}) ⊗ t(Π_{v_{2}})q_{v_{0}}^{−s}]^{−1} ,
L(s, Π_{v_{0}} ⊗ μ_{v_{0}}, Asai) = det [I − t(Π_{v_{1}} ⊗ μ_{v_{1}}) ⊗ t(Π_{v_{2}} ⊗ μ_{v_{2}})q_{v_{0}}^{−s}]^{−1} .

Our assertion follows from the fact that μ_{v_{1}}μ_{v_{2}} = 1.

The following result is then true in the context of GL(n) with n odd:

Proposition 1: Let Π be a cuspidal automorphic representation of GL(n) with n odd. Suppose that the central character ω of Π is trivial on F_{𝔸}^{×} (that is, is distinguished). Then Π̃ ≃ Π^{σ} if and only if Π is distinguished by H.

Proof: We first remark that the relation Π̃ ≃ Π^{σ} implies ωω^{σ} = 1. This is equivalent to ω|F_{𝔸}^{×} = 1 or ω|F_{𝔸}^{×} = ω_{E/F}.

For now consider an arbitrary cuspidal representation Π of GL(n, E) with central character ω. Let ω_{H} be the restriction of ω to F_{𝔸}^{×}. Suppose that ω_{H} = 1 and Π is distinguished. Then ([F2]) Π̃ ≃ Π^{σ}.

Conversely, assume that Π is a cuspidal automorphic representation of GL(n). It is a result of [F1] and [FZ] that the partial Asai L-function has a pole at s = 1 if and only if ω_{H} = 1 and Π is distinguished. Suppose that Π̃ ≃ Π^{σ} and ω_{H} = 1. Then the partial Rankin–Selberg L-function has a pole at s = 1. From the previous lemma it follows that either the partial Asai L-function attached to Π has a pole at s = 1 or the partial Asai L-function attached to Π ⊗ μ has a pole at s = 1. But the restriction of the central character of Π ⊗ μ to F_{𝔸}^{×} is ω_{E/F} ≠ 1. Thus the Asai L-function attached to Π has a pole and Π is distinguished.

Remark 1: By the results of Shahidi on L-functions attached to Eisenstein series, the Asai L-functions do not vanish at s = 1. We could use this to give a global proof (rather than a local proof as in [F2]) of the only if part of the proposition.

Remark 2: Likewise, for n even, then Π satisfies Π̃ = Π^{σ} if and only if either Π is distinguished or Π ⊗ μ is distinguished. In particular, the central character of Π is then distinguished; this gives a positive answer to a question raised in [Ro1]. On the other hand, according to the discussion in [F2], in the first case Π should be the unstable base change of a stable cuspidal packet; in the second case, it should be the stable base change of a stable cuspidal packet.

If Π is distinguished we will need to express the linear form

J (ϕ) := ∫_{Z_{H}(F_{𝔸})H(F)\H(F_{𝔸})} ϕ(h)dh

on the space of Π as a product of local linear forms. When the real places of F split in E, the uniqueness theorem of [F2] implies the existence of such a decomposition. We will however give an explicit decomposition without appealing to the uniqueness result. To that end, we review the construction of [F1]. Let Φ be a Schwartz–Bruhat function on F_{𝔸}^{n} and ϕ a vector in Π. Set

W (g) = ∫ ϕ(ng)\overline{θ}(n)dn.

We set

Ψ(s, W, Φ) := ∫_{N(F_{𝔸})\H(F_{𝔸})} W (ϵh)ϕ[(0, 0, …, 1)h] | det h |^{s} dh.

The rational diagonal matrix ϵ = diag(ϵ_{1}, ϵ_{2}, …, ϵ_{n}) ∈ A(E) is so chosen that tr(ϵ_{i}/ϵ_{i+1}) = 0, for every i. The integrand is indeed invariant under N(F_{𝔸}). We assume that W and Φ are product of local functions. We choose S_{0} sufficiently large so that outside S the function W_{v} is K_{v}-invariant (equal to 1 on K_{v}) and outside S_{0}, Φ_{v_{0}} is the characteristic function of O_{v_{0}}^{n}. Moreover, ϵ is in K_{v} for v ∉ S. If v_{0} is inert in E and v is the corresponding place of E, we introduce a local integral, which is absolutely convergent for ℜs ≥ 1, and, in fact, in a half strip ℜs > 1 − ϵ ([F2], [FZ]):

Ψ(s, W_{v}, Φ_{v_{0}}) := ∫_{N(F_{v_{0}} )\H(F_{v_{0}} )} W_{v}(ϵh)Φ_{v_{0}}[(0, …, 0, 1)h] | det h |^{s} dh.

If v_{0} splits into two places v_{1}, v_{2} then we introduce a local integral which converges for ℜs ≥ 1 and, in fact, in a half strip ℜs > 1 − ϵ ([JS]),

Ψ(s, W_{v_{1}}, W_{v_{2}}, Φ_{v_{0}}) :=

∫_{N(F_{v_{0}} )\H(F_{v_{0}} )} W_{v_{1}}(ϵ_{v_{1}}h)W_{v_{2}}(ϵ_{v_{2}}h)Φ_{v_{0}}[(0, …, 0, 1)h] | det h |^{s} dh.

Let S_{i} be the set of places in S_{0} which are inert in E. Likewise, let S_{s} be the set of places in S_{0} which split in E. Then ([F1])

∫ ϕ(h) ( ∑_{ξ∈F^{n}−0} Φ(ξh)) | det h |_{F}^{s} dh = Ψ(s, W, Φ)

= L^{S_{0}}(s, Π, Asai) × ∏_{v_{0}∈S_{i}} Ψ(s, W_{v}, Φ_{v_{0}}) ∏_{v_{0}∈S_{s}} Ψ(s, W_{v_{1}}, W_{v_{2}}, Φ_{v_{0}}).

Taking the residue at s = 1, we get

cJ (ϕ) ∫ Φ(x)dx =Res_{s=1}L^{S_{0}}(s, Π, Asai)

× ∏_{v_{0}∈S_{i}} Ψ(1, W_{v}, Φ_{v_{0}}) ∏_{v_{0}∈S_{s}} Ψ(1, W_{v_{1}}, W_{v_{2}}, Φ_{v_{0}}),

where the constant c > 0 depends only on the choice of the Haar measures. For v_{0} ∈ S_{i}, let us set

J_{v_{0}}(W_{v}) := ∫_{N(n−1,F_{v_{0}} )\GL(n−1,F_{v_{0}} )} W_{v}(ϵh)dh.

Here we embed GL(n − 1) into GL(n) as a factor of the Levi-subgroup of type (n − 1, 1). The integral is absolutely convergent ([F2], [FZ]). It is clear that the linear form J_{v_{0}} is invariant under P_{1}(F_{v_{0}}). We claim that it is actually invariant under GL(3, F_{v_{0}}). Indeed

Ψ(1, W_{v}, Φ_{v_{0}}) = ∫_{K_{v_{0}}} ∫_{F_{v_{0}}^{×}} J_{v_{0}}(ρ(k)W_{v})Φ_{v_{0}}[(0, 0, …, t)k] | t |^{n} d^{×}tdk.

On the other hand, for a suitable choice of the Haar measures,

∫ Φ_{v_{0}}(x)dx = ∫_{K_{v_{0}}} ∫_{F_{v_{0}}^{×}} Φ_{v_{0}}[(0, 0, …, t)k] | t |^{n} d^{×}tdk.

It follows that there is a linear form γ on the Whittaker model of π_{v} such that

∫_{K_{v_{0}}} ∫_{F_{v_{0}}^{×}} J_{v_{0}}(ρ(k)W_{v})Φ_{v_{0}}[(0, 0, …, t)k] | t |^{n} d^{×}tdk
= γ(W_{v}) ∫_{K_{v_{0}}} ∫_{F_{v_{0}}^{×}} Φ_{v_{0}}[(0, 0, ⋯, t)k] | t |^{n} d^{×}tdk,

for any W_{v} and any function Φ_{v_{0}}. This implies that for any smooth function f on K_{v_{0}} which is invariant under K_{v_{0}} ∩ P_{1,v_{0}} on the left

∫_{K_{v_{0}}} ∫_{F_{v_{0}}^{×}} J_{v_{0}}(ρ(k)W_{v})f(k)dk = γ(W_{v}) ∫_{K_{v_{0}}} f(k)dk.

The same relation is then true for any smooth function f on K_{v_{0}}. It follows that γ = J_{v_{0}} and that J_{v_{0}} is invariant under K_{v_{0}}. Thus it is in fact invariant under GL(n, F_{v_{0}}). Likewise, if v_{0} splits into v_{1}, v_{2} then an invariant linear form on Π_{v_{1}} ⊗ Π_{v_{2}} is given by

J_{v_{0}}(W_{v_{1}} ⊗ W_{v_{2}}) := ∫_{N(n−1,F_{v_{0}} )\GL(n−1,F_{v_{0}} )} W_{v_{1}}(ϵ_{1}h)W_{v_{2}}(ϵ_{2}h)dh.

In fact this is known (see [B], [Ba], and the next section). For a suitable positive constant c′, we get

c′J (ϕ) = Res_{s=1}L^{S_{0}}(s, Π, Asai)) ∏_{v_{0}∈S_{i}} J_{v_{0}}(W_{v}) ∏_{v_{0}∈S_{s}} J_{v_{0}}(W_{v_{1}} ⊗ W_{v_{2}}).

Note that the global linear form J is continuous for the topology of the smooth vectors on the space of the representation Π. In a precise way, let H be the Hilbert space of square integrable elements of Π and V be the space of smooth vectors in H. Let K′ be a compact open subgroup of G^{∞}, then the representation of G_{∞} on H^{K′} is a finite multiple of Π_{∞} and V^{K′} is the space of smooth vectors of this representation. The topology of V^{H′} is the topology defined by the seminorms ϕ ↦ | ρ(X)ϕ| _{2}. The space V is the inductive limit of the spaces V^{K′}. The linear form I is continuous on each one of the spaces V^{K′}, or, what amounts to the same, continuous on V. Likewise for an infinite place v_{0} ∈ S_{i} which is below a place v of E, the linear form I_{v_{0}} may be viewed as a continuous linear form on the space of smooth vectors of Π_{v}. For an infinite place v_{0} ∈ S_{s} which split into v_{1}, v_{2}, the linear form I_{v_{0}} may be viewed as a continuous linear form on the space of smooth vectors of Π_{v_{1}} ⊗ Π_{v_{2}}. Indeed this follows at any rate from the above formula. We define the relative global Bessel distribution J_{Π} as follows: if f is a smooth function of compact support, K-finite on both sides, then we set

J_{Π}(f) = ∑_{i} J (π(f)v_{i})\overline{W(v_{i})},

where v_{i} is an orthonormal basis of H whose elements are K-finite vectors in V. More precisely, let V^{∗} be the topological dual vector space of V. We have thus continuous inclusions with dense image

V ⊂ H ⊂ V^{∗}.

The restriction of the pairing ⟨•, •⟩ of H × H to V × V extends to a sesquilinear form denoted in the same way on the product V × V^{∗} or the product V^{∗} × V. With this notation we may write W(v) = ⟨v, W⟩ and then

⟨Π(n)v, W⟩ = θ(n)⟨v, W⟩, ⟨v, W⟩ = \overline{⟨W, v⟩},

and the above formula reads

J_{Π}(f) = ∑_{i}⟨π(f)v_{i}, J ⟩⟨W, v_{i}⟩.

We still denote by π the representation of G(E_{𝔸}) on V^{∗}. If f is a smooth function of compact support on G(E_{𝔸}) then Π(f)(V^{∗}) ⊂ V. If f is K-finite on both sides, we have

J_{Π}(f) = ⟨W, π(f^{∗})J ⟩ = ⟨π(f)W, J ⟩.

For arbitrary f we take this to be the definition of J_{Π}(f).

For v_{0} ∈ S_{i}, we introduce similarly a local relative local Bessel distribution B_{v_{0}}. For f_{v_{0}} K_{v_{0}}-finite it is given by

B_{v_{0}}(f) = ∑_{ϕ} J_{v_{0}}(Π_{v_{0}}(f)ϕ)\overline{W_{v_{0}}(ϕ)}

where the sum is over an orthonormal basis. There is a similarly defined distribution at the places in S_{s}. Assume f = f_{S}f^{S} where f^{S} is the characteristic function of K^{S}. We find then that the global distribution J_{Π}(f) can be written

J_{Π}(f) = C(Π) ∏_{v_{0}∈S_{0}} B_{v_{0}}(f_{v_{0}}),

where the constant C(Π) can be computed explicitly. Unfortunately, we have not established the identity (18) for all functions f. Thus we cannot use the previous decomposition to obtain the corresponding decomposition of the right hand side of (18), as we would like to.

3. Local relative Bessel distributions Now we let E/F be a local quadratic extension. We set P_{1} = P (E)Z(E) and H_{1} = H(F)Z(F). The elements {w, w_{1}, w_{2}, e} form a set of representatives for the orbits of B(E) on 𝔖(F). Let Ω_{w} be the set of s such that s_{1,3} ≠ 0, Ω_{1} (resp. Ω_{2}) the set of s ∈ 𝔖(F) such that s_{1,2} ≠ 0 or s_{1,3} ≠ 0 (resp. s_{1,3} ≠ 0 or s_{2,3} ≠ 0). These sets are open. For s ∈ 𝔖(F) we have

\overline{s}_{1,3} = \frac{D_{1}}{det s}, D_{1} = | \begin{matrix}s_{1,2} & s_{2,2} \\ s_{1,3} & s_{2,3} \end{matrix} | .

Thus for s ∈ Ω_{w} we have also D_{1} ≠ 0. It follows that s can be written in the form s = b_{1}wb_{2} with b_{i} ∈ B(E). In turn this implies that s = bw\overline{b}^{−1} with b ∈ B(E). Thus Ω_{w} is the orbit of w under B(E). We note that for s ∈ 𝔖(F) the relations s_{1,3} = 0 and s_{1,2} ≠ 0 imply D_{1} = 0 and then s_{2,3} = 0. It easily follows that O_{1} := Ω_{1} − Ω_{w} = P_{1}(E) ∩ 𝔖(F) is the orbit of w_{1} under B(E). Likewise O_{2} := Ω_{2} − Ω_{w} is the orbit of w_{2} under B(E). Finally 𝔖(F) is the union of the open sets Ω_{∗} and the closed set O_{e} := B(E) ∩ 𝔖(F), which is in fact the orbit of e under B(E). The orbit of w under P_{1} contains w and w_{1}ww_{1} = w_{2} but not w_{1}. It follows that Ω_{2} is the orbit of w under P_{1}.

We consider a unitary irreducible representation π of GL(3) on a Hilbert space H. We denote by V the space of smooth vectors in H. We assume that π is distinguished by H(F) in the sense that there is a non-zero continuous linear form J on V such that J (π(h)v) = J (v) for all v ∈ V and all h ∈ H(F). We denote by ω the central character of π; it is necessarily trivial on F^{×}, that is, distinguished. We denote in the same way the extension of ω to H_{1} which is trivial on H(F). Here, it is more convenient to let ψ be a non-trivial character of E trivial on F and define a character θ of N(E) by θ(n) = ψ(n_{2,1} + n_{2,3}). We assume that π is generic and let W be a continuous non-zero linear form on V such that W(π(n)v) = θ(n)W(v), for all n ∈ N(E) and all vectors v. We define the corresponding relative Bessel distribution B as follows: if f is a smooth function of compact support, K-finite on both sides, then

B(f) = ∑_{i} J (π(f)v_{i})\overline{W(v_{i})},

where v_{i} is an orthonormal basis of H whose elements are K-finite vectors in V. As before, we let V^{∗} be the topological dual vector space of V. We have thus continuous inclusions with dense image:

V ⊂ H ⊂ V^{∗}.

The restriction of the pairing ⟨•, •⟩ of H × H to V × V extends to a sesquilinear form denoted in the same way on the product V × V^{∗} or the product V^{∗} × V. We still denote by π the representation of G(E) on V^{∗}. If f is a smooth function of compact support on G(E) then π(f)(V^{∗}) ⊂ V. If f is K-finite on both sides, we have

B(f) = ⟨W, π(f^{∗})J ⟩ = ⟨π(f)W, J ⟩.

For arbitrary f we take this for definition of B(f). Here and below we use an integral notation for the value of B on a function f. The distribution B has the following properties of invariance: for h ∈ H_{1} and n ∈ N(E),

∫ f(hxn)dB(x) = θ(n)^{−1}ω(h)^{−1} ∫ f(x)dB(x).

In this section, we prove the following result:

Theorem 1: The restriction of B to the open set Ω_{F} := P^{−1}(Ω_{w}) is non-zero.

3.1 Proof: First step. We prove the theorem when E/F is the extension ℂ/ℝ. The proof in the non-Archimedean case is similar but simpler. We assume that the restriction of B to Ω_{F} is zero and prove that B is then zero. This will give a contradiction and prove the theorem. Since B is invariant on the left under H(F) it determines a distribution Ξ on 𝔖(F): if Φ is a smooth function of compact support on 𝔖(F) then Ξ(Φ) = B(f) where f is any smooth function of compact support on G(E) such that

Φ(P(g)) = ∫_{H(F)} f(hg)dh.

Thus Ξ vanishes on the set Ω_{w}. We first show that it vanishes on Ω_{1} and Ω_{2}.

To that end, we recall, in the form appropriate to us, some results of [KV]. Let H be a Lie group operating on the right on a manifold X and let H′ be a closed normal subgroup of H. We denote by σ the action of H on X. We also denote by σ the corresponding left action of G on functions on X:

σ(h)Φ(x) = Φ(xσ(h)).

In our case H′ = Z(ℂ)N(ℂ) and H = B(ℂ). In particular, for our purposes, we may assume that H′ and all of its closed subgroups are unimodular. Let θ be a one dimensional character of H′. Suppose that O is a closed orbit of H in X. Let T be a distribution on x, transforming according to θ under H′ and supported on O. Thus

T (σ(h′)Φ) = θ(h′)^{−1}T (Φ).

Let M^{(r)}(O) or simply M^{(r)} be the r-th graded component of the transverse jet bundle of O. Thus M^{(1)} is the normal tangent bundle and M^{(r)} is its r-th symmetric power. Then ([KV]) the support of T is contained in the set of y ∈ O such that for some r the fiber M_{y}^{(r)} contains a non-zero vector transforming under the character θ restricted to H′_{y}, the stabilizer of y in H′.

Going back to the proof of our theorem, we assume that the distribution Ξ vanishes on the open set Ω_{w}. We apply the just recalled results to the group H = B(ℂ) and the subgroup H′ = Z(ℂ)N(ℂ). If y is any point of 𝔖(F) the stabilizer of y in H′ is the product of Z(ℝ) and the stabilizer N_{y} of y in N(ℂ). As a matter of fact Z(ℝ) operates trivially on 𝔖(F). We first show that the restriction of Ξ to Ω_{1} vanishes. Since Ω_{1} = O_{1} ∪ Ω_{w} and O_{1} is closed in Ω_{1} this will follow from the results we recalled above and the following lemma:

Lemma 2: If y ∈ O_{1} then the restriction of θ to N_{y} is non-trivial and the action of N_{y} on M_{y}^{(r)} is unipotent.

Proof: Since B(ℂ) is transitive on O_{1} it will suffice to prove the two assertions for the point y = w_{1}. Then N_{y} is the group of matrices of the form

n = ( \begin{matrix}1 & 0 & x \\ 0 & 1 & \overline{x} \\ 0 & 0 & 1 \end{matrix} ) , x ∈ ℂ.

The first assertion follows at once. For the second assertion, we remark that the action of N_{y} on M_{y}^{1} is algebraic, hence unipotent. Thus the same is true of the action on M_{y}^{(r)}. (In fact, one can check that the action on M_{y}^{1} is trivial.)

The lemma being established it follows that the restriction of Ξ to Ω_{1} is trivial. Likewise the restriction to Ω_{2} is trivial. Equivalently, the restriction of B to Ω = P^{−1}(Ω_{2}) is trivial.

3.2 Proof: Second step. We first recall some results on generic representations of GL(3). We denote by W(π) the space spanned by the functions g ↦ W(π(g)v), with v ∈ V. It is convenient to identify V with W(π).

Lemma 3: For every W ∈ W(π) the restriction of W to P (E) is square integrable modulo N(E). Moreover, the invariant (positive definite) scalar product on W(π) may be taken to be

⟨W_{1}, W_{2}⟩ = ∫_{N(E)\P (E)} W_{1}\overline{W}_{2}(p)d_{r}p.

In particular, the map W ↦ W |P is injective.

Proof: In the non-Archimedean case this is a result of Bernstein valid for GL(n) ([B]). In the Archimedean case, the result follows from the global theory if the representation is a component of a cusp form: the proof is similar to the proof given in the previous section for the invariance of I. At any rate, the result also follows from [JS] and a recent work of Baruch ([Ba]).

Lemma 4: Let U or U(π) be the space of W ∈ W(π) such that the function W |P (E) has compact support modulo N(E). Then the space U is non-zero.

Proof: In the non-Archimedean case, this is a result of Gelfand and Kazhdan ([GK]). The proof in the Archimedean case is similar once we observe the following: let V be a commutative unipotent subgroup of GL(3, E), so that V is isomorphic to a real vector space as a real Lie group; then for every Schwartz function ϕ on V and every smooth vector u in the space of π the vector

π(ϕ)u := ∫_{V} ϕ(v)π(v)udv

is still a smooth vector.

By definition

⟨W, π(f^{∗})W⟩ = ∫ W (g)f(g)dg.

Let W_{0} be an element of U and and let ϕ_{0} be a smooth function of compact support on P (E) such that

∫ ϕ_{0}(np)\overline{θ}(n)dn = W_{0}(p).

Let also ϕ_{1} be a smooth function of compact support on H_{1} such that ∫_{H_{1}} ϕ_{1}(h)ω(h)dh ≠ 0. Since the map (p, h_{1}) ↦ pη^{−1}h_{1}^{−1} is submersive with image Ω_{F}^{−1}, there is a smooth function of compact support f, supported on Ω_{F}, such that for any continuous function Ψ on G(E),

∫ Ψ(g)f(g^{−1})dg = ∫ Ψ(pη^{−1}h_{1}^{−1})ϕ_{0}(p)ϕ_{1}(h_{1})d_{r}pdh_{1}.

In particular, for any W ∈ W(π),

⟨Π(f)W, W ⟩ = ∫ \overline{W} (g)f(g^{−1})dg

= ∫_{H_{1}} ∫_{P} \overline{W} (pη^{−1}h^{−1})ϕ_{0}(p)d_{r}pϕ_{1}(h)dh = ∫ ⟨W_{0}, Π(η^{−1}h^{−1})W ⟩ϕ_{1}(h_{1})dh_{1}

= ⟨∫ Π(h)ϕ(h)dhΠ(η)W_{0}, W ⟩.

Equivalently

Π(f)W = ∫ Π(h)ϕ_{1}(h)dhΠ(η)W_{0},

and thus

B(f) = ∫ ω(h)ϕ_{1}(h)dh⟨W_{0}, J ⟩.

Now B(f) = 0. By our choice of ϕ_{1} we find

⟨Π(η)W_{0}, J ⟩ = 0.

This relation takes place for all W_{0} ∈ U. Since U is invariant under P (E) we have also

⟨π(g)W_{0}, J ⟩ = 0

for all g ∈ Ω_{F}. Since Ω_{F} is dense and the representation Π on V is topologically irreducible, we conclude that J = 0, a contradiction.

3.3 A result on local distinguished representations. The following proposition was part of the thesis of Youngbin Ok, written under the direction of H. Jacquet. It is not really needed in this paper, since, as noted, the result can be established by global means for the components of a distinguished cuspidal representation. Nonetheless, it is interesting that it can be established by purely local means in the context of GL(n, E), where E/F is a quadratic extension of local non-Archimedean fields. It is analogous to the main result of [B].

Proposition 2: Let π be a generic unitary irreducible representation of GL(n, E), distinguished by GL(n, F). We identify the space V of smooth vectors to the Whittaker model of π and denote by J a non-zero continuous linear form on V invariant under GL(n, F). We assume that the generic character θ is trivial on N(F). Let P be the subgroup of matrices with last row (0, 0, …, 0, 1) and N the group of upper triangular matrices with unit diagonal. Then there is a constant c ≠ 0 such that

J (W) = cλ(W),

where

λ(W) := ∫_{N(F)\P (F)} W (p)dp,

and dp is a right invariant measure on N(F)\P (F).

Proof: Indeed, according to [B], page 82, the central exponents of the derivatives π^{[k]} of π, 1 ≤ k < n, are strictly positive. This implies ([F1]) that λ is defined by a convergent integral. Clearly, the linear form λ is invariant under P_{1}(F) = P (F)Z(F). It will suffice to show that it is invariant under G(F). It amounts to the same to show that the distribution Ξ defined by

Ξ(f) = ∑_{i} J (π(f)v_{i})\overline{λ(v_{i})}

is invariant on the right under G(F). A priori, it is invariant on the left under G(F) and on the right under P_{1}(F). As before, it defines a distribution Ξ_{𝔖} on 𝔖(F), invariant under P_{1}(F). The following lemma implies that Ξ_{𝔖} and thus Ξ is invariant under G(F).

Lemma 5: Any distribution on 𝔖(F) which is invariant under P_{1}(F) is also invariant under G(F).

We sketch a proof of this lemma. By the results of [B], the analogous assertion is true for the tangent space of 𝔖(F), that is, the space 𝔖_{0} of matrices X ∈ M(n × n, E) such that X + \overline{X} = 0. Let U ⊂ 𝔖_{0} be the open set of X such that det(1 + X). det(1 − X) ≠ 0. Suppose that μ is a distribution on U invariant under P_{1}(F). Then for any smooth function of compact support f on F^{×} the product of μ and the function f(det(1 + X). det(1 − X)) is the restriction to U of a distribution μ_{f} on 𝔖_{0}. The distribution μ_{f} is supported on U and invariant under P_{1}(F). Thus μ_{f} is invariant under G(F). It follows that μ is invariant under G(F). This being so, for each λ ∈ E^{×} with λ\overline{λ} = 1, let Ω_{λ} be the open set of s ∈ 𝔖(F) such that det(s + λ1_{n}) ≠ 0. The set Ω_{λ} is invariant under G(F) and the map

ι_{λ}(s) := \frac{s − λ}{s + λ}

is an analytic bijection of Ω_{λ} onto U, which commutes to the action of G(F). Now if μ is a distribution on 𝔖(F) invariant under P_{1}(F), then for each λ the distribution ι_{λ}^{∗}(μ|Ω_{λ}) is invariant under G(F). The same is true of μ|Ω_{λ}. Since the open sets Ω_{λ} cover 𝔖(F) our assertion follows.

4. Complements on orbital integrals

We first prove in the context of GL(n) an elementary result which was used without proof in [J2] and [Mao2] in the case of GL(3). Assume that E/F is an unramified quadratic extension of non-Archimedean local fields. Suppose that the residual characteristic is not 2. Denote by ϖ a fixed prime element in the ring of integers O_{F}. Denote by \overline{E}/\overline{F} the residual quadratic extension. Finally, let G = GL_{n}(E) and, as before, let 𝔖 be the symmetric space of elements x ∈ G satisfying x\overline{x} = 1. Let K = GL_{n}(O_{E}), let A be the diagonal subgroup, and w = (δ_{i,n+1−i}) or w_{n} be the standard representative of the longest element in the Weyl group W.

Proposition 3: Every K orbit in 𝔖 contains a unique element of the form aw with a = diag(ϖ^{m_{1}}, …, ϖ^{m_{n}}), m_{1} ≥ m_{2} ≥ ⋯ ≥ m_{n}.

Proof: Our assertion being trivial for n = 1, we may assume n > 1 and our assertion true for n′ < n. By the Cartan decomposition, we may write x = k_{1}ak_{2} where k_{j} ∈ K and a is a diagonal matrix with entries ϖ^{m_{1}}, …, ϖ^{m_{n}}. The element a is uniquely determined up to conjugation by an element of the Weyl group. Furthermore, a = \overline{a} and we may assume that

(21)  m_{1} ≥ m_{2} ≥ ⋯ ≥ m_{n}.

Now k_{1}ak_{2} = \overline{k_{2}}^{−1}a^{−1}\overline{k_{1}}^{−1}. The uniqueness of the Cartan decomposition implies that a^{−1} = u^{−1}au for some u in the Weyl group W. Our hypothesis (21) implies that we may take u = w. Thus we can write x = k_{1}awk_{2}, with k_{i} ∈ K, aw ∈ 𝔖. Our task is then to show that we can choose k_{1}, k_{2} so that k_{2} = \overline{k}_{1}^{−1}.

Suppose first that a = 1, that is, x ∈ K. Our assertion amounts to saying that x = k\overline{k}^{−1} with k ∈ K. Let x′ be the class of x modulo ϖO_{E}. Then x′ = g_{1}.\overline{g}_{1}^{−1}, g_{1} ∈ GL(n, \overline{E}). If k_{1} is congruent to g_{1} we thus have x = k_{1}s_{1}\overline{k}_{1}^{−1} where s_{1} ≡ 1_{n} (mod ϖO_{E}). Thus k_{2} = s_{1} +1 is in K and k_{2}\overline{k}_{2}^{−1} = s_{1}. This proves our assertion in this case.

We pass to the general case where a ≠ 1, that is, m_{1} > 0. We let r ≥ 1 be the integer such that

m_{1} = m_{2} = ⋯ = m_{r} > m_{r−1}.

Since we are only interested in K-orbits, we may assume that k_{2} = 1. Then a^{−1}k_{1}a = w\overline{k}_{1}^{−1}w is in K. Let Q be the standard parabolic subgroup of type (r, n − 2r, r) if n > 2r and type (r, r) if n = 2r. Denote by M its standard Levi-factor, U its unipotent radical and V the unipotent radical of the opposite parabolic subgroup. The above relation implies that k_{1} is congruent to an element of MV. Thus we can write k_{1} = v_{1}mu_{1} with v_{1} ∈ V ∩ K, m ∈ M ∩ K and u_{1} ∈ U ∩ K. We then have x = v_{1}mawv_{2} with v_{2} ∈ V. The uniqueness of this decomposition implies that v_{2} = \overline{v}_{1}^{−1}. We are thus reduced to the case where x = maw. If n > 2r then

x = ( \begin{matrix}m_{1} & 0 & 0 \\ 0 & m_{2} & 0 \\ 0 & 0 & m_{3} \end{matrix} ) ( \begin{matrix}a_{1} & 0 & 0 \\ 0 & a_{2} & 0 \\ 0 & 0 & a_{1}^{−1} \end{matrix} ) ( \begin{matrix}0 & 0 & w_{r} \\ 0 & w_{n−2r} & 0 \\ w_{r} & 0 & 0 \end{matrix} )

with m_{1}, m_{3} ∈ GL(r, O_{E}), m_{2} ∈ GL(n − 2r, O_{E}). Set

k = ( \begin{matrix}m_{1} & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ) .

Then x is equal to

k ( \begin{matrix}a_{1} & 0 & 0 \\ 0 & a_{2} & 0 \\ 0 & 0 & a_{1}^{−1} \end{matrix} ) ( \begin{matrix}0 & 0 & w_{r} \\ 0 & w_{n−2r} & 0 \\ w_{r} & 0 & 0 \end{matrix} ) \overline{k}^{−1}.

The conclusion follows from the induction hypothesis applied to the element

m_{2}a_{2}w_{n−2r}

of the symmetric sapce for GL(n − 2r). The case n = 2r is similar.

To prove the convergence of the local and global orbital integrals, as well as the convergence of the geometric part of our relative trace formulas we use the following lemma, that we prove in the context of GL(n):

Lemma 6: Let E be a local field and Ω be a compact set of GL(n, E). Let A be the group of diagonal matrices, N the group of upper triangular matrices with unit diagonal. The relations n_{i} ∈ N(E), a ∈ A(E),

tn_{1}an_{2} ∈ Ω

imply that det a is in a compact set of E^{×}, the entries of a in a compact set of E, and n_{1}, n_{2} in compact sets of N(E). If E is non-Archimedean, let K = GL(n, O_{E}). Then the relations a ∈ A(E), n_{i} ∈ N(E),

tn_{1}an_{2} ∈ K, a ∈ A ∩ K

imply that n_{1} and n_{2} are in K.

Proof: We prove the second assertion. The proof of the first assertion is similar. We recall that the leading principal minors of a matrix are invariant of the action of N(E) × N(E) on GL(n, E). Thus under the assumptions of the Lemma the leading principal minors of the matrix are units. Our lemma amounts to say that if x is a matrix with integral entries and all its leading principal minors are units (including the determinant) then x =^{t} n_{1}an_{2} with n_{i} ∈ N ∩ K and a ∈ A ∩ K. Inductively, it suffices to prove that such an x can be written in the form

x = ( \begin{matrix}1_{n−1} & 0 \\ u & 1 \end{matrix} ) ( \begin{matrix}g & 0 \\ 0 & 1 \end{matrix} ) ( \begin{matrix}1_{n−1} & v \\ 0 & 1 \end{matrix} )

with g ∈ GL(n − 1, O_{E}) and u, v integers. Indeed, the right hand side can be written in the form

( \begin{matrix}g & gv \\ ug & ∗ \end{matrix} ) .

Thus, at any rate, x can be written in this form and det g is a unit. In addition g is integral thus g is indeed in GL(n − 1, O_{E}). Finally gv and ug are integers. Thus the same is true of u and v and our assertion follows.

The lemma can be applied to expressions of the form n_{1}wan_{2} and easily implies the desired convergence of the orbital integrals, local and global. The global integrals are product of local integrals, all absolutely convergent and almost all equal to one. In addition, for a given function, only finitely many of the global integrals are non-zero.

5. Estimates for functionals To prove that the relative trace formula is absolutely convergent, it will be convenient to use estimates for the global Whittaker linear forms and period integrals discussed in the introduction. The estimates in question are easily obtained in the case of GL(2) and GL(3). In the context of GL(n) it would not be possible to obtain these estimates and one would have to use more sophisticated methods.

5.1 Integral representation of a section. In this subsection and the two next ones, we discuss the case of the Whittaker linear form and the period integral for GL(2). Thus we let G be the group GL(2). To that end we consider a pair of idele class characters σ = (σ_{1}, σ_{2}) of E. As usual, the characters are assumed to be trivial on the subgroup ℝ_{+}^{×} of E_{𝔸}^{×}. We let I(σ) be the space of smooth functions ϕ: G(E_{𝔸}) → ℂ such that

ϕ [( \begin{matrix}a_{1} & x \\ 0 & a_{2} \end{matrix} ) g] = σ_{1}(a_{1})σ_{2}(a_{2})ϕ(g).

We let K be the standard maximal compact subgroup of G(E_{𝔸}) and set

| ϕ| ^{2} = ∫_{K} | ϕ(k) |^{2} dk.

For a given λ = (λ_{1}, λ_{2}) ∈ ℂ^{2} we set

e^{⟨λ+ρ,H(g)⟩} =| a_{1} |^{1/2+λ_{1}}| a_{2} |^{−1/2+λ_{2}}

if

g = ( \begin{matrix}a_{1} & x \\ 0 & a_{2} \end{matrix} ) k, k ∈ K.

We have a fiber bundle of representations I_{λ,σ}. As usual, we identify the space of the representation I_{λ,σ} to I(σ). Then for ϕ ∈ I(σ), functions of the form

ϕ(g; λ) = ϕ(g)e^{⟨λ+ρ,H(g)⟩}

are the standard sections of the fiber bundle. We let f be a smooth function of compact support on GL(2, E_{𝔸}) and consider the convolution section I_{λ,σ}(f)ϕ(g; λ) defined by

(22)  (I_{λ,σ}(f)ϕ) (g)e^{⟨λ+ρ,H(g)⟩} = ∫ ϕ(gx)e^{⟨λ+ρ,H(gx)⟩}f(x)dx.

We first give an integral representation for (22). This is a variation on old ideas which is also described in [CJ]. For the convenience of the reader we go again through the construction. We assume that f = f_{S}f^{S} where f^{S} is the characteristic function of K^{S}. Then if we set

ϕ_{1}(g) = ∫_{K^{S}} ϕ(gk)dk

we see that ∥ ϕ_{1} ∥≤∥ ϕ ∥ and I_{λ,σ}(f)ϕ = I_{λ,σ}(f_{S})ϕ_{1}. The value of our section on g ∈ G_{S} is thus given by

∫_{G_{S}} ϕ_{1}(gx)e^{⟨λ+ρ,H(gx)⟩}f_{S}(x)dx.

We set (with f̌(g) = f(g^{−1}))

f_{1}(g) := ∫_{K_{S}} ϕ_{1}(k)f̌(k^{−1}g)dk.

Thus f_{1} is a smooth function of compact support and (22) is equal to

∫_{E_{S}×E_{S}^{×}} f_{1} [n ( \begin{matrix}a_{1} & 0 \\ 0 & a_{2} \end{matrix} ) g]

× | a_{1} |^{−1/2−λ_{1}} σ_{1}(a_{1})^{−1} | a_{2} |^{1/2−λ_{2}} σ_{2}(a_{2})^{−1}d^{×}a_{1}d^{×}a_{2}dn

where

n = ( \begin{matrix}1 & x \\ 0 & 1 \end{matrix} ) , dn = dx.

Now for each λ, there is a function Φ_{S}[•; λ] on E_{S}^{2} such that, for g ∈ G_{S},

∫_{E_{S}×E_{S}^{×}} f_{1} [n ( \begin{matrix}a_{1} & 0 \\ 0 & 1 \end{matrix} ) g] | a_{1} |^{−1/2−λ_{1}} σ_{1}(a_{1})^{−1}d^{×}a_{1}dn

= Φ_{S}[(0, 1)g; λ]σ_{1}(det g) | det g |^{1/2+λ_{1}} .

Since the function f_{1} has compact support, the support of the function Φ_{S}[•; λ] is contained in a compact set of E_{S}^{2} − 0 which is independent of λ. Then, for g ∈ G_{S},

(23) I_{λ,σ}(f)ϕ(g; λ) =

σ_{1}(det g) | det g |^{1/2+λ_{1}} ∫_{E_{S}^{×}} Φ_{S}[(0, t)g; λ]σ_{1}σ_{2}^{−1}(t) | t |^{1+λ_{1}−λ_{2}} d^{×}t.

To have a formula valid for all g we introduce the characteristic function Φ_{v} := char(O_{v}^{2}) for v ∉ S and set

Φ[(x, y); λ] := Φ_{S}[(x_{S}, y_{S}); λ] ∏_{v∉S} Φ_{v}[(x_{v}, y_{v})].

Then, for g ∈ G(F_{𝔸}),

(24) I_{λ,σ}(f)ϕ(g; λ) =

\frac{σ_{1}(det g) | det g |^{1/2+λ_{1}}}{L^{S}(λ_{1} − λ_{2} + 1, σ_{1}σ_{2}^{−1})} ∫_{E_{𝔸}^{×}} Φ[(0, t)g; λ]σ_{1}σ_{2}^{−1}(t) | t |^{1+λ_{1}−λ_{2}} (t)d^{×}t.

We need to control Φ[•; λ] in terms of f. We take f in a fixed bounded set B. This means that f = f_{S}f^{S} as before, with S fixed; the function f_{S} has support in a fixed set Ξ, open and relatively compact. We assume that K_{S}Ξ = ΞK_{S} = Ξ; the function f_{S} is invariant on the right and the left under a compact open normal subgroup K′ of ∏_{v∈S,finite} K_{v}. Thus f is invariant under the subgroup K” = K′K^{S} of K^{∞}. For each element X of the enveloping algebra of G_{∞} there is a constant C_{X} such that

sup | ρ(X)f |≤ C_{X}.

We need not consider those σ for which I_{λ,σ}(f) = 0. Thus we may assume that the characters σ_{i} are unramified outside S and that, for v ∈ S and finite, the restrictions of the characters σ_{i,v} to O_{v}^{×} are in a finite set. The Schwartz–Bruhat function Φ_{S}[•; λ] is compactly supported: its support is contained in a fixed (i.e. independent of (λ, σ)) open set of E_{S}^{2} − 0, relatively compact in E_{S}^{2} − 0. The function is invariant on the right under a the action of a fixed compact open subgroup K′_{0} of G_{S}. Thus its support is contained in a fixed compact set of E_{S}^{2} and it is invariant under translation by a compact open subgroup of E_{S_{finite}}. Take λ in a strip a ≤ ℜλ ≤ b. Then Φ_{S}[•; λ] is bounded uniformly in λ and σ. Consider a differential operator ξ on E_{∞}^{2} with constant coefficients. On the complement of (0, 0) we can write ξ = ∑_{i} c_{i}X_{i,a} where the X_{i} are elements of the enveloping algebra, X_{i,a} are the corresponding vectors fields on F_{∞}^{2} and the functions c_{i} are smooth functions on E_{∞}^{2} − (0, 0). A polynomial on the set of characters of E_{𝔸}^{×} (or E_{∞}^{×}) is a function of the form

χ ↦ ⟨χ, X⟩

where X belongs to the enveloping algebra of E_{∞}^{×}. It follows that there is a polynomial P_{ξ}(λ, σ_{1}) such that

| ξΦ_{S}[(x, y); λ] |≤∥ ϕ ∥ |P_{ξ}(λ, σ_{1})|.

In particular, for a given σ_{1}, and λ in a strip, the functions Φ_{S}[(x, y); λ, σ_{1}] remain in a bounded set of C_{c}^{∞}(E_{S}^{2}) and a fortiori of the Schwartz–Bruhat space S(E_{S}^{2}). Their partial Fourier transform Φ̂_{S} with respect to the second variable (see below) are thus also in a bounded set. Moreover, we have in fact more information: for v finite in S, the projection of the support of the function Φ̂_{S} on E_{v}^{2} is contained in a fixed compact set and the function is invariant under a fixed compact open subgroup of E_{v}^{2}. Finally, for every ξ and every integer N there is a majorization of the form

| ξΦ̂_{S}[(x, y); λ] |≤ (1 + | (x_{∞}, y_{∞})| ^{2})^{−N} |P_{ξ,N}(σ_{1}, λ)|.

5.2 Whittaker linear form. We define a linear form W_{λ,σ} on the space I(σ) by

W_{λ,σ}(ϕ) := ∫ ϕ [w ( \begin{matrix}1 & x \\ 0 & 1 \end{matrix} )] e^{⟨λ+ρ,H(∗)⟩}ψ(−x)dx

where w is given by (6). It will be convenient for the rest of this section to assume that λ_{1} = −λ_{2} and to write λ for the pair (λ, −λ). The integral converges for ℜλ >> 0. It is known that it extends by analytic continuation to the whole complex plane (this follows in fact from the forthcoming considerations). What we need are estimates for W_{λ,σ} (I_{λ,σ}(f)ϕ). Consider again a bounded set B. In what follows, we consider only those σ such that I_{λ,σ}(f) ≠ 0 for f ∈ B.

If we use (23) and change variables, we obtain, for ℜλ >> 0,

(25) W_{λ,σ}(I_{λ,σ}(f)ϕ) = \frac{1}{L^{S}(2λ + 1, σ_{1}σ_{2}^{−1})} ∫ Φ̂[(t, t^{−1}); λ] | t |^{2λ} σ_{1}σ_{2}^{−1}(t)d^{×}t

where Φ̂ denotes the Fourier transform with respect to the second variable:

Φ̂(u, v) = ∫ Φ(u, x)ψ(−xv)dx.

This expression converges for all λ. Assuming that f = f_{S}f^{S} as before, we can also write this formula in the form

(26) W_{λ,σ}(I_{λ,σ}(f)ϕ) =

\frac{1}{L^{S}(2λ + 1, σ_{1}σ_{2}^{−1})} ∫_{F_{S}^{×}} Φ̂_{S}[(t, t^{−1}); λ] | t |^{2λ} σ_{1,S}σ_{2,S}^{−1}(t)d^{×}t.

Another way to look at the formula is as follows: Let us assume that ϕ = ⊗ϕ_{v} where ϕ_{v} is in the local induced representation I(σ_{v}) and ϕ_{v} is equal to 1 on K_{v} for v ∉ S. Then for f = f_{S}f^{S}, f_{S} = ⊗_{S}f_{v}, we get

W_{λ,σ}(I_{λ,σ}(f)ϕ) = \frac{1}{L^{S}(2λ + 1, σ_{1}σ_{2}^{−1})} ∏_{v∈S} W_{λ,σ_{v}}(I_{λ,σ_{v}}(f_{v})ϕ_{v})

where the local linear forms W_{λ,σ_{v}}, v ∈ S, have similar integral representations:

(27)  W_{λ,σ_{v}}(I_{λ,σ_{v}}(f_{v})ϕ_{v}) = ∫ Φ̂_{v}[(t, t^{−1}); λ] | t |^{2λ} σ_{1,v}σ_{2,v}^{−1}(t)d^{×}t;

here the function Φ_{v} is determined by ϕ_{v}, f_{v} and σ_{1,v}. It will be convenient to introduce in the local case or global case the representation r_{2} of GL(2) on the space of Schwartz–Bruhat functions in two variables defined by

r_{2}(g)Φ̂ = ĝΦ, gΦ(x, y) = Φ[(x, y)g].

Then

W_{λ,σ_{v}}(I_{λ,σ}(g)I_{λ,σ_{v}}(f_{v})ϕ_{v}) =
∫ r_{2}(g)Φ̂_{v}[(t, t^{−1}); λ] | t |^{2λ} σ_{1,v}σ_{2,v}^{−1}(t)d^{×}tσ_{1}(det g) | det g |^{1/2+λ_{1}} .

This being so, it is easy to obtain majorizations for the integral

∫_{F_{S}^{×}} Φ̂_{S}[(t, t^{−1}); λ] | t |^{2λ} σ_{1,S}σ_{2,S}^{−1}(t)d^{×}t

in a vertical strip. We have then estimates for the derivatives as well. Alternatively, for a given S, we may introduce a normalized functional W_{λ}^{1} on the space of K^{S}-invariant vectors. It is defined by

W_{λ}(ϕ) = \frac{1}{L^{S}(2λ + 1, σ_{1}σ_{2}^{−1})}W_{λ}^{1}(ϕ).

The preceeding discussion gives majorizations for W_{λ}^{1}(I_{λ}(f_{S})ϕ):

Proposition 4: Given a bounded set B there is a polynomial P (λ, σ) such that, for all f ∈ B and all λ with ℜλ = 0,

|W_{λ,σ}^{1}(I_{λ,σ}(f)ϕ)| ≤ |P (λ, σ)|| ϕ| .

Moreover, given f and ϕ, the derivatives of λ ↦ W_{λ,σ}^{1}(I_{λ,σ}(f)ϕ) are of slow increase on the line ℜλ = 0.

Later, we will use estimates for L-functions that we state as a lemma.

Lemma 7: Let S be a finite set of places containing all the places at infinity. For v finite in S let Ξ_{v} be a finite set of characters of O_{v}^{×}. Let Ξ be the set of idele class characters χ of E which are trivial on ℝ_{+}^{×}, are unramified outside S and such that, for v finite in S, the restriction of χ_{v} to O_{v}^{×} belongs to Ξ_{v}. Then there is a polynomial P (t, χ) such that for χ ∈ Ξ and t real

|\frac{1}{L^{S}(1 + it, χ)}| ≤ |P (t, χ)|,

and all χ non-trivial

|L^{S}(1 + it, χ)| ≤ |P (t, χ)|.

Moreover, all derivatives of the functions

t ↦ \frac{1}{L^{S}(1 + it, χ)} and t ↦ L^{S}(1 + it, χ)

are of slow increase.

For a fixed χ, the proof is standard (see, for instance, [L]). The proof can be extended to the general case.

Combining the two previous lemmas, we obtain majorizations for the linear forms W_{λ,σ}.

5.3 Intertwining period integral. We now assume that E = F [\sqrt{δ}]. We assume that the two characters σ_{1} and σ_{2} are related by σ_{2}(a) = σ_{1}(\overline{a})^{−1}. Then σ_{1}σ_{2}^{−1}(a) = σ_{1}|F ∘ Norm. We let T be the torus of GL(2, F) whose elements have the form

t = ( \begin{matrix}a & bδ \\ b & a \end{matrix} ) .

We let η ∈ GL(2, E) be an element such that

ηT η^{−1} = \{( \begin{matrix}a & 0 \\ 0 & \overline{a} \end{matrix} )\}

and consider the intertwining period integral (cf. [JLR])

J(ξ, ϕ; λ, σ) := ∫_{T (F_{𝔸})\GL(2,F_{𝔸})} ϕ(ηh)e^{⟨λ+ρ,H(ηh)⟩}dh.

The integral converges for ℜλ >> 0. As before if we use the integral representation (24) we obtain a formula for J(ξ, I_{λ,σ}(f)ϕ; λ, σ):

(28) J(ξ, I_{λ,σ}(f)ϕ; λ, σ) :=

\frac{1}{L^{S}(2λ + 1, σ_{1}|F ∘ Norm)} × ∫_{GL(2,F_{𝔸})} Φ[(0, 1)ηh; λ]σ_{1}(det h) | det |_{F}^{1+2λ} dh.

Equivalently, we define a Schwartz–Bruhat function Ψ[•; λ] on F_{𝔸}^{2} (which depends on λ) in the following way:

Ψ[(a_{1}, a_{2}); λ] := ∫_{K∩GL(2,F_{𝔸})} ∫_{F_{𝔸}} Φ [(0, 1)η ( \begin{matrix}a_{1} & x \\ 0 & a_{2} \end{matrix} ) k; λ] dxσ_{1}(det k)dk.

Then

(29) J(ξ, I_{λ,σ}(f)ϕ; λ, σ) = \frac{1}{L^{S}(2λ + 1, σ_{1}|F ∘ Norm)}

× ∫_{(F_{𝔸}^{×})^{2}} Ψ[(a_{1}, a_{2}); λ]σ_{1}(a_{1}) | a_{1} |_{F}^{2λ} σ_{1}(a_{2}) | a_{2} |_{F}^{2λ+1} d^{×}a_{1}d^{×}a_{2}.

If we denote by˜ the Fourier transform with respect to the first variable and use the Tate functional equation we get

(30) J(ξ, I_{λ,σ}(f)ϕ; λ, σ) = \frac{1}{L^{S}(2λ + 1, σ_{1}|F ∘ Norm)}

× ∫ Ψ̃[(a_{1}, a_{2}); λ]σ_{1}^{−1}(a_{1}) | a_{1} |_{F}^{1−2λ} σ_{1}(a_{2}) | a_{2} |_{F}^{2λ+1} d^{×}a_{1}d^{×}a_{2}.

In fact, with the previous notations and assumptions, we have

Ψ̃ = Ψ̃_{S_{0}} ∏_{v∉S_{0}} char(O_{v_{0}}^{2})

where Ψ_{S_{0}} is a suitable Schwartz–Bruhat function. Moreover

L^{S}(2λ + 1, σ_{1}|F ∘ Norm) = L^{S_{0}}(2λ + 1, σ_{1}|F)L^{S_{0}}(2λ + 1, ω_{E/F} σ_{1}|F).

Thus we can get a more explicit formula:

(31) \frac{L^{S_{0}}(1 − 2λ, σ_{1}^{−1}|F)}{L^{S_{0}}(1 + 2λ, ω_{E/F} σ_{1}|F)}

× ∫ ∫_{F_{S_{0}}} Ψ̃_{S_{0}}[(a_{1}, a_{2}); λ]σ_{1,S_{0}}^{−1}(a_{1}) | a_{1} |_{F}^{1−2λ} σ_{1,S_{0}}(a_{2}) | a_{2} |_{F}^{2λ+1} d^{×}a_{1}d^{×}a_{2}.

In particular, the above expression is a meromorphic function of λ; on the line ℜλ = 0, the only singularity is a simple pole at λ = 0, if the restriction of σ_{1} to F is trivial. This expression can be used to construct a linear form (depending meromorphically on λ) on the space I(σ) (smooth vectors). Indeed, let us denote (31) by β(f, ϕ; λ). Thus λ ↦ β(f, ϕ; λ) is a meromorphic function of λ with values in the space of bilinear forms on C_{C}^{∞}(G(E_{𝔸})) × I(σ). It satisfies the identity

β(f_{1}, I_{λ,σ}(f_{2})ϕ; λ) = β(f_{1} ∗ f_{2}, ϕ; λ),

because for ℜλ >> 0 the bilinear form is given by a convergent integral and the equality is then obvious. The construction shows that for a given λ the bilinear form is separately continuous. In fact if f = f_{S}f^{S} and ϕ is K^{S} invariant so that ϕ = ϕ_{S}ϕ^{S}, we may write

β(f, ϕ; λ) := \frac{L^{S_{0}}(1 − 2λ, σ_{1}^{−1}|F)}{L^{S_{0}}(1 + 2λ, ω_{E/F} σ_{1}|F)} × β_{S}(f_{S}, ϕ_{S}; λ)

where λ ↦ β_{S}(f_{S}, ϕ_{S}; λ) is a meromorphic function with values in the space of bilinear forms on C_{C}^{∞}(G(E_{S})) × I(σ_{S}) given by

β_{S}(f_{S}, ϕ_{S}; λ) =
∫ ∫_{F_{S_{0}}} Ψ̃_{S_{0}}[(a_{1}, a_{2}); λ]σ_{1,S_{0}}^{−1}(a_{1}) | a_{1} |_{F}^{1−2λ} σ_{1,S_{0}}(a_{2}) | a_{2} |_{F}^{2λ+1} d^{×}a_{1}d^{×}a_{2}.

We use this to prove a lemma:

Lemma 8: Suppose there are n elements ϕ_{i}, 1 ≤ i ≤ n of I(σ_{S}) and n smooth functions of compact support f_{i} on G_{S} such that

∑_{1≤i≤n} I_{λ,σ_{S}}(f_{i})ϕ_{i} = 0;

then

∑_{1≤i≤n} β_{S}(f_{i}, ϕ_{i}; λ) = 0.

Proof: Let f_{α} be an approximation of unity on the group G_{S}. We have

∑ β_{S}(f_{α}, ϕ_{i}; λ) = β_{S} (f_{α}, ∑ I_{λ,σ}(f_{i})ϕ_{i}; λ) = 0.

If we let α tend to infinity, we get f_{α} ∗ f_{i} → f_{i}; the separate continuity of β_{S} implies the lemma.

We may then define a linear form J_{S}(ξ, •; λ, σ) on I(σ_{S}) in the following way: given ϕ, we use the Dixmier Malliavin lemma to write

ϕ = ∑ I_{λ,σ_{S}}(f_{i})ϕ_{i}

and then set

J_{S}(ξ, ϕ; λ, σ) = ∑ β(f_{i}, ϕ_{i}; λ).

By the lemma, the right hand side depends only on ϕ so that the linear form is well defined. In other words, the linear form is uniquely determined by the condition that

J_{S}(ξ, I_{λ,σ_{S}}(f_{S})ϕ; λ, σ) = β(f_{S}, ϕ; λ).

For a given λ it is bounded on any bounded set of I(σ_{S}). The functions in a bounded set of I(σ_{S}) are invariant under a fixed open subgroup of K_{v}, for each finite v ∈ S; for each element X of the enveloping algebra of K_{∞} there is a constant C_{X} such that

| ρ(X)ϕ(k) |≤ C_{X}.

We take for granted that we can write the functions in a bounded set in the form

ϕ = ∑_{1≤i≤n} I_{λ,σ}(f_{i})ϕ_{i},

with n fixed, each f_{i} in a bounded set and each ϕ_{i} in a bounded set. Since

J_{S}(ξ, ϕ; λ, σ) = ∑_{i} β_{S}(f_{i}, ϕ_{i})

our assertion follows. As a matter of fact, one can prove that the above decomposition is possible with highly differentiable functions rather than C^{∞} functions (cf. [A1], Corollary 4.2); this is good enough for the above considerations and the assertion follows.

It will be convenient to introduce a normalized intertwining period integral J^{1}. For ϕ invariant under K^{S}, and f as before, we have

J(ξ, I_{λ,σ}(f)ϕ; λ, σ) = \frac{L^{S_{0}}(1 − 2λ, σ_{1}^{−1}|F)}{L^{S_{0}}(1 + 2λ, ω_{E/F} σ_{1}|F)} × J^{1}(ξ, I_{λ,σ}(f_{S})ϕ; λ, σ).

It is then easy to obtain majorizations:

Proposition 5: The normalized period J^{1} has no singularity on the line ℜλ = 0. Let B be a bounded set of smooth functions of compact support. Then, there is a polynomial P (λ, σ_{1}) such that for any f in B, any ϕ ∈ I(σ), and any σ_{1} whose restriction to F is non-trivial,

|J^{1}(ξ, I_{λ,σ}(f)ϕ; λ, σ)| ≤ | P (λ, σ_{1})| ∥ ϕ ∥ .

Likewise, any derivative of λ ↦ J^{1}(ξ, I_{λ,σ}(f)ϕ; λ, σ) on the line ℜλ = 0 is of slow increase.

Similar estimates are available for J in the case where σ_{1} has a trivial restriction to F.

5.4 Intertwining operator. We can use the same ideas to obtain majorizations for the matrix coefficients of the intertwining operator, that is, the functions of the form

⟨M(w, λ)I_{λ,σ}(f)ϕ, ϕ′⟩

with ϕ ∈ I(σ) and ϕ′ ∈ I(wσ). Indeed

M(w, λ)I_{λ,σ}(f)(g; λ) = ∫ I_{λ,σ}(f)ϕ(wn; λ)dn.

Using (24) and a change of variables, we get this is the inverse of the appropriate L-factor times

∫ Φ[(t, x)g; λ] | det t |^{xλ} σ_{1}σ_{2}(t)^{−1}d^{×}tσ_{1}(det g) | det g |^{λ+1/2} .

We introduce the symplectic Fourier transform

Φ̂(x, y) = ∫ ∫ Φ(u, v)ψ(xv − yu)dudv.

Using the Tate functional equation we get this is

∫ Φ̂[(0, t)g; λ] | det t |^{1−2λ} σ_{2}σ_{1}(t)^{−1}d^{×}tσ_{2}(det g) | det g |^{−λ+1/2} .

Finally, with the previous notation and assumptions, we get, for g ∈ G_{S},

M(w, λ)I_{λ,σ}(f)(g; λ) = \frac{L^{S}(1 − 2λ, σ_{2}σ_{1}^{−1})}{L^{S}(1 + 2λ, σ_{1}σ_{2}^{−1})}

× ∫ Φ̂_{S}[(0, t)g; λ] | det t |^{1−2λ} σ_{2,S}σ_{1,S}^{−1}(t)d^{×}tσ_{2,S}(det g) | det g |^{−λ+1/2} .

It is then easy to derive the required majorizations.

5.5 Whittaker functional for GL(3). In the remainder of this section, we let G be the group GL(3). Let σ = (σ_{1}, σ_{2}, σ_{3}) be a triple of idele class characters of E. They are normalized by the condition that they be trivial on ℝ_{+}^{×}. We consider the space I(σ) of smooth functions ϕ such that

ϕ [( \begin{matrix}a_{1} & x & z \\ 0 & a_{2} & y \\ 0 & 0 & a_{3} \end{matrix} ) g] = σ_{1}(a_{1})σ_{2}(a_{2})σ_{3}(a_{3})ϕ(g).

As usual, we have a holomorphic fiber bundle of induced representations I_{λ,σ}. Here we let λ be a triple of complex numbers (λ_{1}, λ_{2}, λ_{3}) (with ∑ λ_{i} = 0). Then we set

e^{⟨λ+ρ,H(g)⟩} =| a_{1} |^{1+λ_{1}}| a_{2} |^{λ_{2}}| a_{3} |^{−1+λ_{3}}

if g = nak and a = diag(a_{1}, a_{2}, a_{3}). If ϕ is in I_{σ}, then the function ϕ(•, λ) defined by

ϕ(g; λ) = ϕ(g)e^{⟨λ+ρ,H(g)⟩}

is a standard section. To a standard section we can associate an Eisenstein series

E(g, ϕ; λ, σ) = ∑ ϕ(γg; λ)e^{⟨λ+ρ,H(γg)⟩}

and then set

W_{λ,σ}(ϕ) = ∫ E(n, ϕ; λ, σ)\overline{θ}(n)dn.

We define in this way a linear form W_{λ,σ} on the space of K-finite vectors of I_{σ}. Equivalently, W_{λ,σ} is obtained by analytically continuing the integral

∫ ϕ(wn; λ)\overline{θ}(n)dn,

where w is given by (6). In fact it is known that the linear form is defined on the space of smooth vectors. If ϕ = ∏_{v} ϕ_{v} where, for v ∉ S, ϕ_{v} is equal to 1 on K_{v}, then we can write

W_{λ,σ}(ϕ) = \frac{1}{∏_{i<j} L^{S}(1 + λ_{i} − λ_{j}, σ_{i}σ_{j}^{−1})} ∏_{v∈S} W_{λ,σ_{v}}(ϕ_{v}),

where W_{λ,σ_{v}} is a local Whittaker linear form: the analytic properties of the local linear form are known (see the references to Casselman, Shahidi and Wallach), in particular, the fact that it is a continuous linear form on the space of smooth vectors. Again, we can introduce a normalized linear form on the space of K^{S}invariant vectors by

W_{λ,σ}(ϕ) = \frac{1}{∏_{i<j} L^{S}(1 + λ_{i} − λ_{j}, σ_{i}σ_{j}^{−1})}W_{λ,σ}^{1}(ϕ).

We need estimates uniform in λ and σ for W_{λ,σ}^{1}(I_{λ,σ}(f)ϕ).

Proposition 6: Let B be a bounded set of the space of smooth functions of compact support. There is a polynomial P (λ, σ) such that for f_{i} ∈ B, i = 1, 2, for any σ, and any ϕ ∈ σ, any λ with ℜλ = 0,

|W_{λ,σ}^{1}(I_{λ,σ}(f_{1} ∗ f_{2})ϕ)| ≤ |P (λ, σ)|| ϕ| .

We can also show that the derivatives of such a function are of slow increase on the space ℜλ = 0. We will sketch a proof of these estimates below. Combining with standard estimates on Abelian L-functions we will obtain similar estimates for W_{λ,σ}.

Now we discuss how to obtain estimates for the local linear forms. Thus for the remaining part of this section we let E be a local field. We have then the local representation I(σ). The proof is again based on an integral representation for a convolution section and the value of the linear form on a convolution section. We will simply write down the integrals in question and will leave out the details of the rest of the proof. We let f_{1}, f_{2} be two smooth functions of compact support. We find an absolutely convergent expression for W_{λ,σ}(I_{λ,σ}(f_{2} ∗ f_{1})ϕ). We first find an absolutely convergent expression for the section I_{λ,σ}(f_{2} ∗ f_{1})ϕ(g; λ). In a precise way, we set

ϕ(g; λ) = ϕ(g)e^{⟨λ+ρ,H(g)⟩},
ϕ_{1}(g; λ) = ∫ ϕ(gx; λ)f_{1}(x)dx, ϕ_{2}(g; λ) = ∫ ϕ_{1}(gx; λ)f_{2}(x)dx.

It will be convenient to introduce a function ϕ_{1}(•, •; λ) on GL(3) × GL(2) such that

ϕ_{1}(g, e; λ) = ϕ_{1}(g; λ),

ϕ_{1} (( \begin{matrix}a_{1} & X \\ 0 & g_{2} \end{matrix} ) g, x; λ) = ϕ_{1}(g, xg_{2}; λ) | a_{1} |^{1+λ_{1}} σ_{1}(a_{1}) | det g_{2} |^{−1/2} .

From the computations on GL(2), it can be seen that there is a Schwartz–Bruhat function Φ_{1}(•, •; λ) on E^{2} × K, where K is the standard maximal compact subgroup of GL(3, E) such that

ϕ_{1}[k, x; λ] =
σ_{2}(det x) | det x |^{1/2+λ_{2}} ∫_{E^{×}} Φ_{1}[(0, g_{1})x; λ]σ_{2}σ_{3}^{−1}(g_{1}) | g_{1} |^{1+λ_{2}−λ_{3}} d^{×}g_{1}.

We have then

ϕ_{2}(g; λ) = ∫ ϕ_{1}(x; λ)f̌_{2}(x^{−1}g)dx.

At this point we use the Iwasawa decomposition for the parabolic subgroup of type (1, 2) to get that ϕ_{2}(g; λ) is equal to

∫ ϕ_{1}(k_{3}, g_{2}; λ)f̌_{2} (k_{3}^{−1} ( \begin{matrix}1 & X \\ 0 & 1_{2} \end{matrix} ) ( \begin{matrix}a_{1}^{−1} & 0 \\ 0 & g_{2}^{−1} \end{matrix} ) g)

×σ_{1}(a_{1}) | a_{1} |^{1+λ_{1}} d^{×}a_{1} | det g_{2} |^{−1/2} dg_{2}.

Next, we define a function Φ_{2}(•, •; λ) on E^{2} × GL(3, E) by

Φ_{2}[(x, y), g; λ] = ∫ Φ_{1}[(x, y), k_{3}; λ]f̌_{2}(k_{3}^{−1}g)dk_{3}.

The projection of the support of this function on the second variable is contained in a fixed compact set. Now there is a function Φ_{3}(•, •; λ) on E^{2} × M(2 × 3, F) such that

Φ_{3} [(x, y), (0, 1_{2})g; λ] σ_{1}(det g_{1}) | det g |^{1+λ_{1}}=

∫ Φ_{2} [(x, y), ( \begin{matrix}1 & X \\ 0 & 1_{2} \end{matrix} ) ( \begin{matrix}a_{1}^{−1} & 0 \\ 0 & 1_{2} \end{matrix} ) g] σ_{1}(a_{1}) | a_{1} |^{1+λ_{1}} d^{×}a_{1}.

In terms of Φ_{3} we can write

(32) ϕ_{2}(g; λ) = σ_{1}(det g) | det g |^{1+λ_{1}}

× ∫ Φ_{3} [(0, g_{1})g_{2}^{−1}, (0, g_{2})g; λ]

σ_{2}σ_{3}^{−1}(g_{1}) | g_{1} |^{1+λ_{2}−λ_{3}} d^{×}g_{1}σ_{1}σ_{2}^{−1}(det g_{2}) | det g_{2} |^{1+λ_{1}−λ_{2}} d^{×}g_{2}.

We can now write that

W_{λ}(I_{λ}(f_{2} ∗ f_{1})ϕ) = ∫ ϕ_{2}(wn)\overline{θ}(n)dn

= ∫ ϕ_{2} [w_{2} ( \begin{matrix}1 & 0 & 0 \\ 0 & 1 & x \\ 0 & 0 & 1 \end{matrix} ) w_{1}w_{2} ( \begin{matrix}1 & 0 & z \\ 0 & 1 & y \\ 0 & 0 & 1 \end{matrix} )] ψ(−x)ψ(−y)dxdy.

Taking these integrals inside the integral representation leads to an inner integral of the form

∫ Φ [(g_{1}, g_{1}x), (g_{2}, g_{2} ( \begin{matrix}z \\ y \end{matrix} )); λ] ψ(−x − y)dxdydz.

We introduce a partial Fourier transform of Φ_{4}:

Φ_{5} [(•, u), ( \begin{matrix}• & • & a \\ • & • & b \end{matrix} ); λ] =

∫ Φ_{4} [(•, x), ( \begin{matrix}• & • & z \\ • & • & y \end{matrix} ); λ] ψ(−xu − za − yb)dydz.

Recall that the right multiplication of elements of E^{2} by elements of GL(2) produces a representation of GL(2) on the space of Schwartz–Bruhat functions on E^{2}; by Fourier transform, it produces a new representation denoted r_{2}. Likewise, multiplication of elements of M(2 × 3, E) by elements of GL(2) on the left produces a right representation on the space of Schwartz–Bruhat functions on M(2 × 3, E); by Fourier transform we get a right representation of GL(2) denoted by l_{2}. Thus we have also right and left representations of GL(2) on the space of Schwartz–Bruhat functions on E^{2} × M(2 × 3, E). We denote them in the same way. We obtain finally

∫ r_{2}(g_{2})^{−1}Φ_{5} [(g_{1}, g_{1}^{−1}), (g_{2},^{t} g_{2}^{−1} ( \begin{matrix}0 \\ 1 \end{matrix} )); λ]

σ_{2}σ_{3}^{−1}(g_{1}) | g_{1} |^{λ_{2}−λ_{3}} d^{×}g_{1}σ_{1}σ_{2}^{−1}(det g_{2}) | det g_{2} |^{λ_{1}−λ_{2}} d^{×}g_{2}.

We use the Iwasawa decomposition to compute the integral over GL(2):

g_{2} = k_{2} ( \begin{matrix}a_{1} & x \\ 0 & a_{2} \end{matrix} ) , dg_{2} = dk_{2}d^{×}a_{1} | a_{2} |^{−1} d^{×}a_{2}.

We introduce a new Schwartz–Bruhat function

Φ_{6} = ∫ r_{2}(k_{2})^{−1}Φ_{5}l(k_{2})dk_{2}.

Then we get

∫ Φ_{6} [(a_{1}^{−1}g_{1}, a_{2}g_{1}^{−1}), ( \begin{matrix}a_{1} & x & 0 \\ 0 & a_{2} & a_{2}^{−1} \end{matrix} ); λ]

σ_{2}σ_{3}^{−1}(g_{1}) | det g_{1} |^{λ_{2}−λ_{3}} d^{×}g_{1}σ_{1}σ_{2}^{−1}(a_{1}a_{2}) | a_{1}a_{2} |^{λ_{1}−λ_{2}} d^{×}a_{1}d^{×}a_{2}ψ(−xa_{1}^{−1})dx.

We introduce finally the Fourier transform Φ of Φ_{6} defined by

Φ [(•, •), ( \begin{matrix}• & a & • \\ • & • & • \end{matrix} ); λ] = ∫ Φ_{6} [(•, •), ( \begin{matrix}• & x & • \\ • & • & • \end{matrix} ); λ] ψ(−xa)dx.

Our last expression is then

(33) W_{λ}(I_{λ}(f_{1} ∗ f_{2})ϕ) =

∫ Φ [(a_{1}^{−1}g_{1}, a_{2}g_{1}^{−1}), ( \begin{matrix}a_{1} & a_{1}^{−1} & 0 \\ 0 & a_{2} & a_{2}^{−1} \end{matrix} ); λ]

σ_{2}σ_{3}^{−1}(g_{1}) | g_{1} |^{λ_{2}−λ_{3}} d^{×}g_{1}σ_{1}σ_{2}^{−1}(a_{1}a_{2}) | a_{1}a_{2} |^{λ_{2}−λ_{3}} d^{×}a_{2}d^{×}a_{3}.

The previous formal manipulations are justified for ℜλ_{1} > ℜλ_{2} > ℜλ_{3}. The above integral converges for all λ. Thus the above formula is actually true for all λ. It can be used to obtain the required majorizations. It could also be used to analytically continue W_{λ}.

6. Limit formula for J_{χ}(f)

We now discuss the continuous part of the spectrum. We follow the notations of [JLR]. Thus we let E/F be a quadratic extension of number fields. For the time being, let G be the group GL(n) regarded as an algebraic group over E and H the group GL(n) regarded as an algebraic group over F. We denote by N the group of upper triangular matrices with unit diagonal and by θ a generic character of N(E_{𝔸}) trivial on N(E). We let f be a smooth function of compact support on G(𝔸). We define as usual a geometric kernel by (2). Let χ be a cuspidal pair, that is, an equivalence class of pairs (M, π) where M is a standard Levi-subgroup of G and π is a cuspidal representation of M. We assume that the restriction of the central character of π to Z(E_{𝔸}) is ω. We let K_{f,χ} be the corresponding kernel component. With this notation,

K_{f}(x, y) = ∑_{χ} K_{f,χ}(x, y).

In addition we define

K_{f,χ}^{0}(x, y) := \frac{1}{n(χ)} ∑_{(M,π)} ∫_{i𝔄_{P}^{∗}} ∑_{ϕ∈B_{P} (π)} E(x, I_{λ}(f)ϕ; λ)\overline{E(y, ϕ; λ)}dλ.

Here dλ is an appropriate Haar measure on i𝔄_{P}^{∗}. The outer sum is over all pairs (M, π) in the class χ. For a given pair, the Eisenstein series at hand is induced from the standard parabolic subgroup P with Levi-factor M and B_{P} (π) is an orthonormal basis for the induced representation. The integer n(χ) is the number of chambers in 𝔞_{P}. Note that the full kernel K_{f,χ} has an analogous expression which involves Eisenstein series built out of residual forms. It follows from the functional equation of the Eisenstein series that, for a given χ, all the kernels

(x, y) ↦ ∫_{i𝔄_{P}^{∗}} ∑_{ϕ∈B_{P} (π)} E(x, I_{λ}(f)ϕ; λ)\overline{E(y, ϕ; λ)}dλ

are actually equal. Thus, if we call n_{1}(χ) the number of pairs in the given class χ, and choose one representative pair (M, π) in the class, we can also write

K_{f,χ}^{0}(x, y) = \frac{n_{1}(χ)}{n(χ)} ∫_{i𝔄_{P}^{∗}} ∑_{\{ϕ\}} E(x, I_{λ}(f)ϕ; λ)\overline{E(y, ϕ; λ)}dλ.

Lemma 9: For all y,

∫ K_{f,χ}(y, n)θ(n)dn = ∫ K_{f,χ}^{0}(y, n)θ(n)dn.

Proof: This follows from the fact that residual Eisenstein series are degenerate ([MW]).

In most of this paper, we will assume that f = f_{1} ∗ f_{2}^{∗} where each function f_{i} is K-finite. Then

K_{f,χ}^{0}(x, y) = \frac{n_{1}(χ)}{n(χ)} ∑ ∫_{i𝔄_{P}^{∗}} ∑_{\{ϕ\}} E(x, I_{λ}(f_{1})ϕ; λ)\overline{E(y, I_{λ}(f_{2})ϕ; λ)}dλ.

We recall some consequences of Arthur’s majorizations ([A1]). Suppose f^{∞} is a smooth function of compact support on the group G^{∞}. Then, there is a continuous semi-norm μ on the space of smooth functions of compact support on G_{∞} such that for any X, Y in the enveloping algebra of G_{∞} and any f of the form f = f_{∞}f^{∞} with f_{∞} K_{∞}-finite,

(34)  ∑_{χ} \frac{n_{1}(χ)}{n(χ)} ∫_{i𝔄_{P}^{∗}} | ∑_{ϕ∈B_{P} (π)} ρ(X)E(x, I_{λ}(f)ϕ; λ)\overline{ρ(Y^{∗})E(y, ϕ; λ)}|dλ
≤ μ(X ∗ f_{∞} ∗ Y) ∥ x ∥^{N}∥ y ∥^{N} .

Here the action of the enveloping algebra is by infinitesimal right translation: if X is in the Lie algebra of G_{∞} and h is a smooth function on G_{∞}, then

ρ(X)h(g) = \frac{dh(gexp(tX)}{dt}|_{t=0} .

Similarly

(35)  ∑_{χ} |ρ_{1}(X)ρ_{2}(Y̌)K_{f,χ}(x, y)| ≤ μ(X ∗ f_{∞} ∗ Y) ∥ x ∥^{N}∥ y ∥^{N} .

Here the index 1, for instance, means that the operator ρ(X) is applied to the function x ↦ K_{f,χ}(x, y). As usual we denote by T the truncation parameter, by d(T) the minimum of the α(T) for α ∈ Δ_{0}. If N is given, we let G(E_{𝔸}, N) be the set of x ∈ G(E_{𝔸}) such that log ∥ x ∥≤ N. We denote by Λ_{m}^{T} the relative truncation operator introduced in [JLR]. We denote by Λ_{1,m}^{T}K(x, y) the partial truncation operator, that is, the truncation operator applied to the function x ↦ K(x, y). We recall a lemma which is based on the work of Arthur (cf. [J3] Proposition 21):

Lemma 10: Let Ω be a compact subset of G(E_{𝔸}). There is a constant C such that if f has support in G(E_{𝔸}, N) then, for d(T) ≥ C(1 + N), y ∈ Ω any x ∈ H(F_{𝔸}) and any χ,

Λ_{1,m}^{T}K_{f}(x, y) = K_{f}(x, y), Λ_{1,m}^{T}K_{f,χ}(x, y) = K_{f,χ}(x, y).

Now let f^{∞} be given. Let Ω be an open subset of G_{∞} relatively compact and such that K_{∞}ΩK_{∞} = Ω. Let also Ω_{1} be a compact subset of G(F_{𝔸}). Then there is C such that for d(T) > C, f = f_{∞}f^{∞}, with f_{∞} supported on Ω, x ∈ H(F_{𝔸}), y ∈ Ω_{1}, and any χ,

Λ_{1,m}^{T}K_{f,χ}(x, y) = K_{f,χ}(x, y).

Since truncation transforms slowly increasing functions into rapidly decreasing functions, we see from (35) that we can truncate the series

∑_{χ} K_{f,χ}(x, y)

termwise. Moreover, there is a continuous semi-norm μ on C_{c}^{∞}(Ω) such that, under the same conditions,

(36)  ∑_{χ} |K_{f,χ}(x, y)| ≤ μ(f_{∞}).

We apply this to a compact set Ω_{1} of N(E_{𝔸}) such that N(E_{𝔸}) = Ω_{1}N(E). After integrating over N(E)\N(E_{𝔸}) we get

∑_{χ} |∫ K_{f,χ}(x, n)θ(n)dn| ≤ μ(f_{∞})

or

∑_{χ} |∫ K_{f,χ}^{0}(x, n)θ(n)dn| ≤ μ(f_{∞}).

At this point we introduce the spectral distributions to be investigated in this paper:

(37)  J_{χ}(f) = ∫_{H(F)Z(F_{𝔸})\H(F_{𝔸})} dh ∫_{N(E)\N(E_{𝔸})} K_{χ}(h, n)θ(n)dn.

By the previous majorization this is well defined. Note that in fact J_{χ}(f) is equal to

∫_{H(F)Z(F_{𝔸})\H(F_{𝔸})} dh ∫_{N(E)\N(E_{𝔸})} K_{χ}^{0}(h, n)θ(n)dn.

From the previous majorization we also get

(38)  ∑_{χ} |J_{χ}(f)| ≤ μ(f_{∞})

for f = f_{∞}f^{∞}, f_{∞} supported on Ω. In particular, for a fixed χ,

(39)  |J_{χ}(f)| ≤ μ(f_{∞}).

This shows that the linear form f_{∞} ↦ J_{χ}(f), defined on K-finite functions, extends to a distribution. Thus J_{χ}(f) is defined for all f. Let S be a finite set of cuspidal data. Consider the inequality

∑_{χ∈S} |J_{χ}(f)| ≤ μ(f_{∞}).

It is true for f_{∞} K-finite with support on Ω. By continuity, it is then true for any f_{∞} ∈ C_{c}^{∞}(Ω). Since S is arbitrary, we see that the above inequality (38) is true for any f_{∞} ∈ C_{c}^{∞}(Ω).

To continue we use Lemma 9 to conclude that there is C > 0 such that, for x ∈ H(F_{𝔸}) and f = f_{∞}f^{∞} with f_{∞} ∈ C_{c}^{∞}, we have, for d(T) > C and any χ,

Λ_{m}^{T} ∫ K_{f,χ}^{0}(x, n)θ(n)dn = ∫ Λ_{1,m}^{T}K_{f,χ}^{0}(x, n)θ(n)dn = ∫ K_{f,χ}^{0}(x, n)θ(n)dn.

It follows that

J_{χ}(f) = ∫ ∫ Λ_{1,m}^{T}K_{f,χ}^{0}(x, n)θ(n)dndx.

Furthermore, the inequality (34) implies that we can take the truncation and the integrations inside the expression for K^{0}. We introduce

W_{λ,σ}(ϕ) := ∫ E(n, ϕ; λ)\overline{θ}(n)dn.

We often drop σ from the notation and write simply W_{λ}(ϕ). Then we get

Proposition 7: Let f_{i}^{∞}, i = 1, 2 be given as well as an open set Ω of G_{∞} relatively compact and bi-K invariant. There is C such that if f_{i} = f_{i,∞}f_{i}^{∞}, i = 1, 2, f_{i,∞} supported on Ω, then

J_{χ}(f) = \frac{n_{1}(χ)}{n(χ)} ∫_{i𝔄_{P}^{∗}} ∑_{\{ϕ\}} (∫ Λ_{m}^{T}E(h, I_{λ}(f_{1})ϕ, λ)dh) \overline{W_{λ}(I_{λ}(f_{2})ϕ)}dλ,

for all T with d(T) ≥ C.

Note that the integral is absolutely convergent in the sense that

∫_{i𝔄_{P}^{∗}} | ∑_{\{ϕ\}} (∫ Λ_{m}^{T}E(h, I_{λ}(f_{1})ϕ, λ)dh) \overline{W_{λ}(I_{λ}(f_{2})ϕ)}|dλ

is finite. We now go back to the case n = 3.

7. Contribution of the parabolic of type (2, 1)

We let χ be a cuspidal pair of the form (M, σ), where M = GL(2) × GL(1) and σ is a cuspidal automorphic representation of GL(2, E_{𝔸}) × GL(1, E_{𝔸}). Then n(χ) = n_{1}(χ). By [JLR], the term J_{χ}(f) is zero unless σ is distinguished, that is, σ = σ_{1} ⊗ σ_{2} where σ_{1} is a cuspidal automorphic representation of GL(2, E_{𝔸}) distinguished by GL(2, F_{𝔸}) and σ_{2} an idele class character of E trivial on F_{𝔸}^{×}. As usual, σ_{1} is normalized by the condition that its central character be trivial on ℝ_{+}^{×}. Likewise for σ_{2}. We let Q be the parabolic subgroup of type (2, 1) and Q′ the parabolic subgroup of type (1, 2). As before we can define the space I(σ) of smooth functions ϕ : G(E_{𝔸}) → ℂ such that, for any k ∈ K, the function m ↦ ϕ(mk) is in the space of smooth vectors of σ. Let ϕ be a K-finite vector in I(σ). Consider an Eisenstein series E(g, ϕ; λ). Then

(40) ∫ Λ_{m}^{T}E(h, ϕ; λ)dh = v_{Q} \frac{e^{⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}J(1, ϕ, 0) + v_{Q′} \frac{e^{⟨sλ,T ⟩}}{⟨sλ, α̌_{Q′}⟩}J(1, M(s, λ)ϕ, 0),

where s = w_{1}w_{2} and

J(1, ϕ, 0) := ∫_{K_{H}} ∫_{M_{H}(F)\M_{H}^{1}} ϕ(mk)dkdm;

here M_{H} is the product of GL(2, F) and GL(1, F) and K_{H} the standard maximal compact subgroup of H(F_{𝔸}). The period integral J(1, •, 0) will be also denoted by J_{0,σ} or J_{0,σ_{1},σ_{2}}. It is a continuous linear form on I(σ). In fact,

⟨sλ, α̌_{Q′}⟩ = ⟨λ, s^{−1}α̌_{Q′}⟩ = −⟨λ, α̌_{Q′}⟩.

Denoting by α̌_{i} the simple coroots, we take T of the form T = tα̌_{1} + tα̌_{2} with t > 0. Then the projections of T on 𝔄_{Q} and 𝔄_{Q′} respectively have the form: T_{Q}α̌_{Q}, T_{Q}α̌_{Q′} with T_{Q} > 0. Thus ⟨sλ, T ⟩ = −⟨λ, T ⟩. Hence the above formula can be written

∫ Λ_{m}^{T}E(h, ϕ; λ)dh = v_{Q} \frac{e^{⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}J(1, ϕ, 0) − v_{Q} \frac{e^{−⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}J(1, M(s, λ)ϕ, 0).

Since both sides of the formula must be holomorphic at λ = 0 we find

(41)  J(1, ϕ, 0) = J(1, M(s, 0)ϕ, 0).

To continue, we set

\begin{matrix}d_{+,ϕ,ϕ′}(λ) = & J(1, ϕ′, 0)\overline{W_{λ}(I_{λ}(f_{2})ϕ)}, \\ d_{−,ϕ,ϕ′}(λ) = & J(1, M(s, λ)ϕ′, 0)\overline{W_{λ}(I_{λ}(f_{2})ϕ)}. \end{matrix}

Then d_{+,ϕ,ϕ′}(0) = d_{−,ϕ,ϕ′}(0) and (42)

J_{χ}(f) = v_{Q} ∑_{ϕ,ϕ′} ∫ [\frac{e^{⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}d_{+,ϕ,ϕ′}(λ) − \frac{e^{−⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}d_{−,ϕ,ϕ′}(λ)] ⟨I_{λ}(f_{1})ϕ′, ϕ⟩dλ.

To compute the limit for T → +∞, we use the method of (G, M) families developed by Arthur ([A3]). Here it amounts to writing the integrand as the sum of the following terms:

e^{⟨λ,T ⟩} \frac{d_{+,ϕ,ϕ′}(λ) − d_{+,ϕ,ϕ′}(0)}{⟨λ, α̌_{Q}⟩}⟨I_{λ}(f_{1})ϕ′, ϕ⟩

−e^{−⟨λ,T ⟩} \frac{d_{−,ϕ,ϕ′}(λ) − d_{+,ϕ,ϕ′}(0)}{⟨λ, α̌_{Q}⟩}⟨I_{λ}(f_{1})ϕ′, ϕ⟩

+ (\frac{e^{⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩} − \frac{e^{−⟨λ,T ⟩}}{⟨λ, α̌_{Q}⟩}) d_{+,ϕ,ϕ′}(0)⟨I_{λ}(f_{1})ϕ′, ϕ⟩.

In the first term, the quotient is a smooth function, even at λ = 0. The reciprocal of the denominator is bounded in the complement of a neighborhood of 0, and ⟨I_{λ}(f)ϕ′, ϕ⟩ is integrable and square integrable. Now J(1, ϕ, 0) is independent of λ. If x and y are in compact sets we know that

∫ | ∑_{ϕ} E(x, I_{λ}(f_{2})ϕ; λ)\overline{E(y, I_{λ}(f_{2})ϕ; λ)}|dλ

is finite. Integrating over N × N we conclude that

∫ ∑_{ϕ} |W_{λ}(I_{λ}(f_{2})ϕ)|^{2} dλ

is finite as well. Thus d_{+,ϕ,ϕ′} is square integrable. Hence the first term can be viewed as the Fourier transform evaluated on the projection of T on 𝔄_{Q} of an integrable function. Thus it tends to zero as T tends to infinity. Since the intertwining operator is bounded, a similar conclusion applies to the second term. Finally, the last term can be computed as the integral of d_{+,ϕ,ϕ′}(0) times the Fourier transform of

⟨I_{λ}(f_{1})ϕ′, ϕ⟩

over the convex hull of the set T_{Q}, −T_{Q}. Thus it tends to ⟨I_{0}(f_{1})ϕ, ϕ′⟩ as T tends to infinity. We arrive at our result:

J_{χ}(f) = v_{Q} ∑_{ϕ,ϕ′}⟨I_{λ}(f)ϕ′ϕ⟩d_{+,ϕ,ϕ′}(0)

or

(43)  J_{χ}(f) = v_{Q} ∑_{\{ϕ\}} J(1, I_{0}(f_{1})ϕ, 0)\overline{W_{0}(I_{0}(f_{2})ϕ)}.

On the space of smooth vectors of the representation I_{0,σ_{1},σ_{2}} we now have two continuous linear forms: the period form J_{0,σ_{1},σ_{2}} and the Whittaker form W_{0,σ_{1},σ_{2}}. Using the notation of generalized vectors we see that we can write our result in the form given in the next proposition.

Proposition 8: For any smooth function of compact support f we have

J_{χ}(f) = v_{Q}⟨I_{0,σ_{1},σ_{2}}(f)W_{0,σ_{1},σ_{2}}, J_{0,σ_{1},σ_{2}}⟩.

We have established the result for a K-finite function. Since both sides are distributions the identity is true for all functions.

8. Contribution of the Borel subgroup

Suppose that χ is represented by the pair (A, σ) where A is the group of diagonal matrices and σ = (σ_{1}, σ_{2}, σ_{3}). As before, the characters are normalized by the condition that they be trivial on the group ℝ_{+}^{×}. As before, λ denotes an element in 𝔄^{∗}. We introduce the complex variables s_{1}, s_{2}, s_{3} defined by

e^{⟨λ,H(a)⟩} =| a_{1} |^{s_{1}}| a_{2} |^{s_{2}}| a_{3} |^{s_{3}} .

We define the space I(σ). Let ϕ be K-finite in I(σ). Consider the corresponding Eisenstein series E(h, ϕ, λ).

According to [JLR], Theorem 40, the period of the truncated Eisenstein series ∫ Λ_{m}^{T}E(h, ϕ, λ)dh is equal to a sum of the terms indexed by the standard parabolic subgroups of G whose Levi-factors are isomorphic to products of the groups GL(1) and GL(2). In our case,

∫ Λ_{m}^{T}E(h, ϕ, λ)dh = I(B, ϕ, λ) + I(Q, ϕ, λ) + I(Q′, ϕ, λ),

where Q and Q′ are the standard parabolic subgroups of type (2, 1) and (1, 2) respectively, while B is, as before, the standard Borel subgroup. The term attached to B vanishes unless the characters σ_{i} are all distinguished. If they are, then

(44)  I(B, ϕ, λ) = v_{B} ∑_{s} \frac{e^{⟨sλ,T ⟩}}{∏_{α∈Δ}⟨sλ, α̌⟩}J(1, M(s, λ)ϕ, 0)

where the sum is over all s in the Weyl group and, as before,

J(1, ϕ, 0) = ∫_{K_{H}} ϕ(k)dk.

The terms attached to Q and Q′ vanish unless σ_{i}(a) = σ_{j}(\overline{a})^{−1} for some pair of indices i, j and the third character σ_{k} is distinguished. Assuming this, the terms are as follows:

(45)  I(Q, ϕ, λ) = ∑_{w} v_{Q} \frac{e^{⟨(wλ)_{Q},T ⟩}}{⟨(wλ)_{Q}, α̌_{Q}⟩}J(ξ_{Q}, M(w, λ)ϕ, (wλ)_{0}^{Q})

and

(46)  I(Q′, ϕ, λ) = ∑_{w} v_{Q′} \frac{e^{⟨(wλ)_{Q′} ,T ⟩}}{⟨(wλ)_{Q′}, α̌_{Q′}⟩}J(ξ_{Q′}, M(w, λ)ϕ, (wλ)_{0}^{Q′}).

Here

ξ_{Q} = w_{1}, η_{Q}.\overline{η}_{Q}^{−1} = ξ_{Q},
ξ_{Q′} = w_{2}, η_{Q′}.\overline{η}_{Q′}^{−1} = ξ_{Q′}.

In (45), the sum is over all w such that w^{−1}α_{1} > 0 and wσ is trivial on the torus T_{Q} defined by T_{Q} = A(E) ∩ η_{Q}H(F)η_{Q}^{−1}, that is, the torus of matrices of the form

diag(α, \overline{α}, β), α ∈ E, β ∈ F.

Likewise in (46), the sum is over all w such that w^{−1}α_{2} > 0 and wσ is trivial on the torus T_{Q′}. The definition of the intertwining period will be recalled below.

Let us write J_{χ}(f) as a sum

J_{χ}(f) = J_{χ}^{min}(f) + J_{χ}^{max}(f)

where

J_{χ}^{min}(f) = \frac{n_{1}(χ)}{n(χ)} ∑_{ϕ} ∫ I(B, I_{λ}(f_{1})ϕ, λ)\overline{W_{λ}(I_{λ}(f_{2})ϕ)}dλ

and

J_{χ}^{max}(f) = \frac{n_{1}(χ)}{n(χ)} ∑_{ϕ} ∫ (I(Q, I_{λ}(f_{1})ϕ, λ) + I(Q′, I_{λ}(f_{1})ϕ, λ)) \overline{W_{λ}(I_{λ}(f_{2})ϕ)}dλ.

8.1 The term J_{χ}^{min}(f). We now compute I(B, ϕ, λ). We assume that the characters σ_{i} are distinguished for i = 1, 2, 3 since otherwise the term vanishes. As before, we assume that f_{1} and f_{2} are themselves convolutions of K-finite functions. In particular,

f_{1} = f_{1}^{1} ∗ f_{1}^{2}.

Then

I_{λ}(f_{1})ϕ = ∑_{ϕ′}⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩ϕ′.

Thus we get

J_{χ}^{min}(f) = v_{B} \frac{n_{1}(χ)}{n(χ)} ∑_{ϕ,ϕ′} ∫ ∑_{s} \frac{e^{⟨sλ,T ⟩}}{∏_{α∈Δ}⟨sλ, α̌⟩}A_{ϕ,ϕ′}(λ)B_{ϕ,ϕ′}(λ)dλ

where we have set

B_{ϕ,ϕ′}(λ) = ⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩,
A_{ϕ,ϕ′}(λ) = J(1, M(s, λ)I_{λ}(f_{1}^{1})ϕ′, 0)\overline{W_{λ}(I_{λ}(f_{2})ϕ)}.

As before, the function λ ↦ B_{ϕ,ϕ′}(λ) is of rapid decrease as well as all its derivatives, while

λ ↦ A_{ϕ,ϕ′}(λ)

is of slow increase as well as all its derivatives.

We shall make use of the theory of (G, M) families as developed by Arthur. Set

c_{s}(λ) = e^{⟨sλ,T ⟩} and d_{s}(λ) = A_{ϕ,ϕ′}(λ).

Let P(A) be the set of parabolic subgroups defined over F and having A as a Levi-factor. If P is such a parabolic, there is s such that sP = B. We set

(47)  c_{P} (λ) = c_{s}(λ), d_{P} (λ) = d_{s}(λ).

We shall need to verify the following lemma.

Lemma 11: The families (c_{P}) and (d_{P}) are (G, A) families in the sense of Arthur.

Proof: This amounts to saying the following: suppose s is a simple reflection and s_{1}, s_{2} are such that ss_{1} = s_{2}; if λ verifies s_{2}λ = s_{1}λ then

c_{s_{2}}(λ) = c_{s_{1}}(λ), d_{s_{2}}(λ) = d_{s_{1}}(λ).

This is clear for the first family. For the second, the relation to be established reads: for s_{2}λ = s_{1}λ,

J(1, M(s_{1}, λ)I_{λ}(f_{1}^{2})ϕ′, 0)\overline{W_{λ}(I_{λ}(f_{2})ϕ)} = J(1, M(s_{2}, λ)I_{λ}(f_{1}^{2})ϕ′, 0)\overline{W_{λ}(I_{λ}(f_{2})ϕ)}.

By linearity this reduces to

J(1, M(s_{1}, λ)ϕ′, 0)\overline{W_{λ}(ϕ)} = J(1, M(s_{2}, λ)ϕ′, 0)\overline{W_{λ}(ϕ)}.

Since M(s_{2}, λ) = M(s, s_{1}λ)M(s_{1}, λ) we see that the relation we have to establish is

J(1, M(s, λ)ϕ′, 0)\overline{W_{λ}(ϕ)} = J(1, ϕ′, 0)\overline{W_{λ}(ϕ)}

if sλ = λ. Say s = w_{1}. There are two cases to consider.

Assume first that σ_{1} ≠ σ_{2}. Since the period is induced from GL(2) it suffices to prove this relation in the context of GL(2). That is, we consider the induced representation I_{λ,σ_{1},σ_{2}} and an Eisenstein series E(g, ϕ, λ; σ_{1}, σ_{2}). Then

∫ Λ_{m}^{T}E(h, ϕ, λ; σ_{1}, σ_{2})dh = v_{B} \frac{e^{⟨λ,T ⟩}}{⟨λ, α̌⟩} ∫ ϕ(k)dk − v_{B} \frac{e^{−⟨λ,T ⟩}}{⟨λ, α̌⟩} ∫ M(σ, λ)ϕ(k)dk.

The holomorphy at λ = 0 implies that, as claimed,

∫ ϕ(k)dk = ∫ M(σ, 0)ϕ(k)dk.

Now suppose that σ_{1} = σ_{2}. The relation w_{1}λ = λ is equivalent to λ_{0}^{Q} = 0. Then

M(w_{2}, λ) = M(s, λ_{0}^{Q}) = M(w_{2}, 0) = −1

(the last relation from the GL(2) theory). Thus we cannot argue as before. However, W_{λ} vanishes for λ_{0}^{Q} = 0 which makes the above relation trivially true. Indeed, we have

W_{λ}(I_{λ}(f)ϕ) = W_{λ}^{1}(I_{λ}(f)ϕ)\frac{1}{∏_{i<j} L^{S}(1 + s_{i} − s_{j}, σ_{i}σ_{j}^{−1})}

where S is a suitable finite set of places. The linear form W_{λ}^{1} is defined locally and holomorphic for ℜλ = 0. Here σ_{1} = σ_{2}. The relation λ_{0}^{Q} = 0 gives s_{1} = s_{2} so that the reciprocal of the L-factors vanishes as claimed. Our assertion follows.

Now, with the notations of Arthur ([A3]),

J_{χ}^{min}(f) = \frac{n_{1}(χ)}{n(χ)} ∑_{ϕ} ∫ ∑_{P ∈P(A)} c_{P} (λ)d_{P} (λ)\frac{1}{θ_{P} (λ)}b(λ)dλ,

where we set b(λ) = ⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩. Following [A3], Lemma 6.3, we rewrite this as

∑_{Q⊃A} c_{A}^{Q}(λ)d′_{Q}(λ).

We recall the definition of c_{A}^{Q}:

c_{A}^{Q}(λ) = ∑_{P ∈P(A),P ⊆Q} c_{P} (λ)θ_{P}^{Q}(λ)^{−1}.

Explicitly, for Q ∈ P(A) we have Q = sB for some s and then

c_{A}^{Q}(λ) = e^{⟨sλ,T ⟩}.

If Q = G then

c_{A}^{G}(λ) = ∑_{s} \frac{e^{⟨sλ,T ⟩}}{⟨sλ, α̌_{1}⟩⟨sλ, α̌_{2}⟩}.

Suppose that Q is a maximal proper parabolic subgroup containing A. Then Q = sQ_{i} with i = 1 or i = 2. Here Q_{1} (resp. Q_{2}) is the parabolic subgroup of type (2, 1) (resp. (1, 2)). Then

c_{A}^{Q}(λ) = \frac{e^{⟨sλ,T ⟩}}{⟨sλ, α̌_{i}⟩} + \frac{e^{⟨s_{i}sλ,T ⟩}}{⟨s_{i}sλ, α̌_{i}⟩}.

We will not need the explicit value of the functions d′_{Q}(λ) except for d′_{G}(λ). In fact d′_{G}(λ) is equal to the common value d(0) of all the functions d_{P} (λ), P ∈ P(A) at the point λ = 0. We will, however, need the fact that each function d′_{Q}(λ) depends only on the projection λ_{Q} on i𝔄_{Q}^{∗} in the decomposition

(48)  𝔄_{0}^{∗} = 𝔄_{0}^{Q∗} ⊕ 𝔄_{Q}^{∗}.

Each function d′_{Q}(λ) is a smooth function on i𝔄_{Q} ([A3], Lemma 6.1). Moreover, it is of slow increase as well as all its derivatives. Indeed this follows from the proof of Lemma 6.1 in [A3] and the following elementary lemma:

Lemma 12: Suppose that ϕ is a smooth function, of slow increase as well as all its derivatives, on ℝ^{n}. If ϕ vanishes on the hyperplane {x|⟨x, α⟩ = 0}, then

ϕ(x) = ⟨x, α⟩ϕ_{0}(x)

where ϕ_{0} is a smooth function, of slow increase as well as all its derivatives.

We recall the proof of the lemma. We can introduce a system of coordinates (x_{1}, x_{2}, …, x_{n}) such that the hyperplane is the coordinate hyperplane x_{1} = 0.

Then we can take

ϕ_{0}(x_{1}, x_{2}, …, x_{n}) = ∫_{0}^{1} \frac{∂ϕ}{∂x_{1}} (tx_{1}, x_{2}, …, x_{n}).

Thus we can write our integral as

J_{χ}^{min}(f) = \frac{n_{1}(χ)}{n(χ)}v_{Q} ∑_{ϕ,ϕ′} ∑_{Q⊃A} ∫ c_{A}^{Q}(λ)d′_{Q}(λ)b(λ)dλ.

Recall that the factor c_{A}^{Q}(λ) depends in fact on T.

Lemma 13: The following limit exists:

lim_{T →∞} ∫ c_{A}^{Q}(λ)d′_{Q}(λ)b(λ)dλ.

If Q = G and the σ_{i} are distinct, then the limit is d′_{G}(0)b(0). Otherwise, that is, if Q ≠ G or if the σ_{i} are not all distinct, then the limit is zero.

Proof: Let χ_{T} be the characteristic function of the convex hull of the points {s^{−1}T } where s runs over W. The function c_{A}^{G}(λ) is the Fourier transform of χ_{T} ([A3], p. 36). Therefore

\begin{matrix}lim_{T →∞} ∫ c_{A}^{G}(λ)d′_{G}(λ)b(λ)dλ = & lim_{T →∞} ∫ χ_{Y} (H)d̂′_{G}b(H)dH \\ = & ∫ χ_{Y} (H)d̂′_{G}b(H)dH \\ = & d′_{G}(0)b(0). \end{matrix}

If the σ_{i} are not distinct, then d′_{G}(0) = 0 since, as seen above, it involves the Whittaker function W_{λ}(I_{λ}(f_{2})ϕ) which vanishes at λ = 0.

Next, suppose that Q = sQ_{i} is a maximal parabolic subgroup containing A. We may decompose T as T = T_{Q_{i}} + T_{0}^{Q_{i}} according to the direct sum in (48). Set T_{Q} = s^{−1}T_{Q_{i}} and T_{0}^{Q} = s^{−1}T_{Q}, and let α = s^{−1}α_{i}. Then

c_{A}^{Q}(λ) = e^{⟨λ,T_{Q}⟩}(\frac{e^{⟨λ,T_{0}^{Q}⟩}}{⟨λ, α̌⟩} − \frac{e^{−⟨λ,T_{0}^{Q}⟩}}{⟨λ, α̌⟩} ),

and

(49)  ∫ c_{A}^{Q}(λ)d′_{Q}(λ)b(λ)dλ

can be written

∫_{𝔄_{Q}^{∗}} e^{⟨λ_{Q},T_{Q}⟩}d′_{Q}(λ_{Q})( ∫_{𝔄_{0}^{Q∗}} (\frac{e^{⟨λ,T_{0}^{Q}⟩}}{⟨λ, α̌⟩} − \frac{e^{−⟨λ,T_{0}^{Q}⟩}}{⟨λ, α̌⟩} )b(λ_{0}^{Q} + λ_{Q})dλ_{0}^{Q})dλ_{Q}.

The inner integral can be written in the form

∫ χ_{T_{0}^{Q}}(H)χ_{T_{0}^{Q}}b̂(H, λ_{Q})dH,

where b̂(•, λ_{Q}) is the Fourier transform of the function λ_{0}^{Q} ↦ b(λ_{0}^{Q} + λ_{Q}) and χ_{T_{0}^{Q}} is the characteristic function of an interval (depending on T_{0}^{Q}). The inner integral represents a Schwartz function of λ_{Q}, depending on the parameter T_{0}^{Q}. In particular, this Schwartz function remains in a fixed bounded set (independent of T_{0}^{Q}) of the space of Schwartz functions. Thus (49) is then the Fourier transform of this function evaluated at T_{Q} and hence tends to zero as T (and hence also T_{Q}) tends to infinity.

The Riemann–Lebesgue Lemma applies directly in the case Q = sB.

Finally, we see that

\begin{aligned}J_{χ}^{min}(f) & = \frac{n_{1}(χ)}{n(χ)}v_{B} ∑_{ϕ,ϕ′} J(1, I_{0}(f_{1}^{1})ϕ′, 0)\overline{W_{0}(I_{0}(f_{2})ϕ)}⟨I_{0}(f_{1}^{2})ϕ, ϕ′⟩ \\ & = v_{B} ∑_{ϕ} J(1, I_{0}(f_{1})ϕ, 0)\overline{W_{0}(I_{0}(f_{2})ϕ)}. \end{aligned}

As before this can be written in terms of generalized vectors. Note that n_{1}(χ) = n(χ) if the σ_{i} are distinct. Also, J_{χ}^{max}(f) = 0 if the σ_{i} are distinguished and distinct.

Proposition 9: Suppose that χ is represented by three characters σ_{i}, 1 ≤ i ≤ 3. The term J_{χ}^{min}(f) vanishes unless the characters are distinct and distinguished. If they are distinguished and distinct, then J_{χ}(f) = J_{χ}^{min}(f) and, for any smooth function of compact support f,

(50)  J_{χ}(f) = v_{B}⟨I_{0,σ}(f)W_{0,σ}, J_{0,σ}⟩.

This identity has been established when f is a K-finite function which is itself a sum of sufficiently many convolution products of K-finite functions. Since both sides are distributions, the equality is in fact true for all f.

8.2 The term J_{χ}^{max}(f). This terms vanishes unless the following condition is satisfied:

(∗) One of the three characters, σ_{i} say, is distinguished and the two others σ_{j} and σ_{k} are related by σ_{j}(a) = σ_{k}(\overline{a})^{−1}.

If either σ_{j} or σ_{k} is distinguished, then σ_{j} = σ_{k}.

According to Proposition 9, J_{χ}^{min}(f) = 0 unless the σ_{j} are distinguished and distinct. It follows that

J_{χ}(f) = J_{χ}^{max}(f)

when (*) is satisfied. Without loss of generality, we may assume that

σ_{1}(a) = σ_{2}(\overline{a})^{−1}, σ_{3} is distinguished.

Let m(χ) = \frac{1}{2} if σ_{1} = σ_{2} and m(χ) = 1 if σ_{1} ≠ σ_{2}.

Proposition 10: Suppose χ is represented by (σ_{1}, σ_{2}, σ_{3}) where σ_{1}(a)=σ_{2}(\overline{a})^{−1} and σ_{3} is distinguished. Then J_{χ}(f) = J_{χ}^{max}(f) and, for all smooth functions of compact support,

J_{χ}(f) = m(χ)v_{Q} ∫ ⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩dλ,

where W_{λ,σ} is as before and J_{λ,σ} is the generalized vector defined below. Moreover, if B is a bounded set, there is C > 0 such that for f ∈ B,

∑_{χ} m(χ) ∫ |⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩| dλ < C,

the sum over all χ of the above type.

To begin the proof, we note that

(51)  I(Q, ϕ, λ) = ∑_{w} v_{Q} \frac{e^{⟨(wλ)_{Q},T ⟩}}{⟨(wλ)_{Q}, α̌_{Q}⟩}J(ξ_{Q}, M(w, λ)ϕ, (wλ)_{0}^{Q})

and

(52)  I(Q′, ϕ, λ) = ∑_{w} v_{Q′} \frac{e^{⟨(wλ)_{Q′} ,T ⟩}}{⟨(wλ)_{Q′}, α̌_{Q′}⟩}J(ξ_{Q′}, M(w, λ)ϕ, (wλ)_{0}^{Q′}).

In (51), the sum is over all w such that w^{−1}α_{1} > 0 and wσ is trivial on the torus T_{η_{Q}}, that is, the torus of matrices of the form

diag(α, \overline{α}, β), α ∈ E, β ∈ F.

Likewise in (52), the sum is over all w such that w^{−1}α_{2} > 0 and wσ is trivial on the torus T_{η_{Q′}}.

We shall need to use the following functional equation for the intertwining period.

Lemma 14: With the previous notations, if λ = λ_{0}^{Q} is in i𝔞_{0}^{Q∗}, then

J(ξ_{Q}, ϕ, λ) = J(ξ_{Q′}, M(w_{1}w_{2}, λ)ϕ, w_{1}w_{2}λ).

Proof: By Proposition 33 of [JLR]

J(ξ_{Q}, ϕ, λ) = J(w_{1}ξ_{Q}w_{1}, M(w_{2}, λ)ϕ, w_{2}λ).

Similarly, if λ′ ∈ i𝔞_{0}^{Q′∗} then

J(ξ_{Q′}, ϕ, λ′) = J(w_{1}ξ_{Q′}w_{1}, M(w_{1}, λ′)ϕ′, w_{1}λ′).

Now

w_{1}ξ_{Q}w_{1} = w_{2}ξ_{Q}w_{2}.

Moreover, if λ′ = w_{1}w_{2}λ and ϕ′ = M(w_{1}w_{2}, λ)ϕ then

w_{2}λ = w_{1}λ′ and M(w_{2}, λ)ϕ = M(w_{1}, λ′)ϕ′.

The lemma follows.

Remark: If σ_{1} and σ_{2} are not distinguished, then we can derive the functional equation from the formula

(53) ∫ Λ_{m}^{T}E(h, ϕ, λ)dh =v_{Q} \frac{e^{⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}J(ξ_{Q}, ϕ, λ_{0}^{Q})

− v_{Q} \frac{e^{−⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}J(ξ_{Q′}, M(w_{1}w_{2}, λ_{0}^{Q} + λ_{Q})ϕ, w_{1}w_{2}(λ_{0}^{Q})).

Using the fact that the singularities on the line λ_{Q} = 0 cancel, we find

(54)  J(ξ_{Q}, ϕ, λ_{0}^{Q}) = J(ξ_{Q′}, M(w_{1}w_{2}, λ_{0}^{Q})ϕ, w_{1}w_{2}λ_{0}^{Q}).

We also note that J(ξ_{Q}, ϕ, λ_{0}^{Q}) is induced from GL(2) in the following sense:

J(ξ_{Q}, ϕ, λ_{0}^{Q}) = ∫_{K_{H}} ∫_{T_{Q}(F_{𝔸})\M_{H}(F_{𝔸})} e^{⟨λ_{0}^{Q}+ρ_{Q},H(η_{Q}m)⟩}ϕ(η_{Q}mk)dmdk

where M_{H} is the Levi subgroup of type (2, 1) in H. Now M_{H} is isomorphic to GL(2, F) times GL(1, F). Thus the integral over M_{H} is really an integral defining the intertwining period for the representation I(σ_{1}, σ_{2}) of GL(2), or more precisely, its analytic continuation (section 5.3). It inherits the analytic properties of the intertwining period for GL(2). In particular, the following result follows directly from the integral representation (31).

Lemma 15: If σ_{1}(a) = σ_{2}(\overline{a})^{−1} but σ_{1} is not distinguished, then

J(ξ_{Q}, ϕ, λ_{0}^{Q})

has no singularity on the set ℜλ_{O}^{Q} = 0. If σ_{1}(a) = σ_{2}(\overline{a})^{−1} and σ_{1} is distinguished, then J(ξ_{Q}, ϕ, λ_{0}^{Q}) has a simple pole at λ_{0}^{Q} = 0.

More precisely, we can use the GL(2) theory to define, for S large enough, a normalized intertwining period J^{1} such that

(55)  J(ξ_{Q}, I_{λ}(f_{1}^{1})ϕ, λ_{0}^{Q}) = J^{1}(ξ_{Q}, I_{λ}(f_{1}^{1})ϕ, λ_{0}^{Q})\frac{L^{S_{0}}(1 − 2s, σ_{1}^{−1}|F)}{L^{S_{0}}(1 + 2s, ω_{E/F} σ_{1}|F)}.

The normalized period J^{1} is defined locally and has no singularity on the line ℜλ = 0.

The corresponding relation for W is

(56) W_{λ}(I_{λ}(f)ϕ) = W_{λ}^{1}(I_{λ}(f)ϕ) × \frac{1}{L^{S_{0}}(1 + 2s, σ_{1}|F)L^{S_{0}}(1 + 2s, ω_{E/F} σ_{1}|F)}

× \frac{1}{∏_{i<j,(i,j)≠(1,2)} L^{S}(1 + s_{i} − s_{j}, σ_{i}σ_{j}^{−1})}.

Suppose first that σ_{1} = σ_{2} = σ_{3}. Then the condition that wσ be trivial on T_{η_{Q}} or T_{η_{Q′}} is vacuous. Thus in (51) the sum is over w ∈ {e, w_{2}, w_{2}w_{1}} and, in (52), the sum is over w ∈ {e, w_{1}, w_{1}w_{2}}. In this case n_{1}(χ)/n(χ) = 1/6. It will be convenient to introduce the following expressions:

A_{1}(λ, ϕ) =v_{Q} \frac{e^{⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}J(ξ_{Q}, ϕ, λ_{0}^{Q})

+ v_{Q′} \frac{e^{⟨(w_{1}w_{2}λ)_{Q′} ,T ⟩}}{⟨(w_{1}w_{2}λ)_{Q′}, α̌_{Q′}⟩}J(ξ_{Q′}, M(w_{1}w_{2}, λ)ϕ, (w_{1}w_{2}λ)_{0}^{Q}),

A_{2}(λ, ϕ) =v_{Q′} \frac{e^{⟨λ_{Q′} ,T ⟩}}{⟨λ_{Q′}, α̌_{Q′}⟩}J(ξ_{Q′}, ϕ, λ_{0}^{Q′})

+ v_{Q} \frac{e^{⟨(w_{2}w_{1}λ)_{Q},T ⟩}}{⟨(w_{2}w_{1}λ)_{Q}, α̌_{Q}⟩}J(ξ_{Q}, M(w_{2}w_{1}, λ)ϕ, (w_{2}w_{1}λ)_{0}^{Q}),

A_{3}(λ, ϕ) =v_{Q} \frac{e^{w_{2}⟨λ_{Q},T ⟩}}{⟨(w_{2}λ)_{Q}, α̌_{Q}⟩}J(ξ_{Q}, M(w_{2}, λ)ϕ, (w_{2}λ)_{0}^{Q})

+ v_{Q′} \frac{e^{⟨(w_{1}λ)_{Q′} ,T ⟩}}{⟨(w_{1}λ)_{Q′}, α̌_{Q′}⟩}J(ξ_{Q′}, M(w_{1}, λ)ϕ, (w_{1}λ)_{0}^{Q}).

The following relations are easily checked:

A_{1}(w_{2}w_{1}λ, M(w_{2}w_{1}, λ)ϕ) = A_{2}(λ, ϕ), A_{1}(w_{2}λ, M(w_{2}, λ)ϕ) = A_{3}(λ, ϕ).

The functional equation of the Eisenstein series implies that

W_{wλ}(M(w, λ)ϕ) = W_{λ}(ϕ).

Thus altogether J_{χ}^{max}(f) is 1/6 times the integral of the sum of the following terms:

A(λ) = ∑_{ϕ} A_{1}(λ, ϕ)\overline{W_{λ}(I_{λ}f_{2})ϕ)},

B(λ) = ∑_{ϕ} A_{1}(w_{2}w_{1}λ, M(w_{2}w_{1}, λ)ϕ)\overline{W_{w_{2}w_{1}λ}(I_{w_{2}w_{1}λ}(f_{2})M(w_{2}w_{1}, λ)ϕ)},

C(λ) = ∑_{ϕ} A_{1}(w_{2}λ, λ, ϕ)\overline{W_{w_{2}λ}(I_{w_{2}λ}(f_{2})(M(w_{2}, λ)ϕ)}.

Since the sums over the orthonormal basis are independent of the choice of the basis, we get in fact

B(λ) = A(w_{2}w_{1}λ), C(λ) = A(w_{2}λ).

We note the simplifying relations

v_{Q′} = v_{Q},
⟨(wλ)_{Q′}, T ⟩ = −⟨λ_{Q}, T ⟩, ⟨(wλ)_{Q′}, α̌_{Q′}⟩ = −⟨λ_{Q}, α̌_{Q}⟩,
(wλ)_{0}^{Q′} = w(λ_{0}^{Q}).

We will show in a moment that A(λ) is a smooth integrable function of λ. After a change of variables, it will follow that J_{χ}^{max}(f) is equal to the integral of \frac{1}{2}A(λ).

If σ_{1} ≠ σ_{3}, then also σ_{2} ≠ σ_{3} and the sum in (51) reduces to a single term, w = e. The sum in (52) reduces to a single term, w = w_{1}w_{2}. Again we will show that A(λ) is a smooth integrable function of λ. Furthermore, n_{1}(χ)/n(χ) = 1/2. Defining A(λ) as before, we will show that A(λ) is a smooth integrable function of λ. It will follow that J_{χ}^{max}(f) is also equal to the integral of \frac{1}{2}A(λ) in this case.

Set

(57)  d_{+,ϕ,ϕ′}(λ) =J(ξ_{Q}, I_{λ}(f_{1}^{1})ϕ′, λ_{0}^{Q})\overline{W_{λ}(I_{λ}(f_{2})ϕ)},
(58)  d_{−,ϕ,ϕ′}(λ) =J(ξ_{Q′}, M(w, λ)I_{λ}(f_{1}^{1})ϕ′, wλ_{0}^{Q})\overline{W_{λ}(I_{λ}(f_{2})ϕ)}.

Then, in all cases where (*) is satisfied, A(λ) is equal to

∑_{ϕ,ϕ′} [\frac{e^{⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}d_{+,ϕ,ϕ′}(λ) − \frac{e^{−⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}d_{−,ϕ,ϕ′}(λ)]⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩.

Lemma 16: The functions

d_{+,ϕ,ϕ′}, d_{−,ϕ,ϕ′}

have no singularities on ℜλ = 0. On ℜλ = 0 they are of slow increase as well as all their derivatives. Furthermore, the relation

d_{+,ϕ,ϕ′}(λ_{0}^{Q}) = d_{−,ϕ,ϕ′}(λ_{0}^{Q})

holds. In particular, A(λ) is a smooth integrable function of λ.

Proof: Lemma (14) implies the relation of the lemma. Then the assertion on A(λ) follows from the first part of the Lemma.

We check the assertions relative to d_{+,ϕ,ϕ′}. Indeed,

d_{+,ϕ,ϕ′}(λ) =J^{1}(ξ_{Q}, I_{λ}(f_{1}^{1})ϕ′, λ_{0}^{Q})\overline{W_{λ}^{1}(I_{λ}(f_{2})ϕ)}

× \frac{1}{|L^{S_{0}}(1 + 2s, ω_{E/F} σ_{1}|F)|^{2}}

× \frac{1}{∏_{i<j,(i,j)≠(1,2)} L^{S}(1 + s_{j} − s_{i}, σ_{j}σ_{i}^{−1})}.

This expression has no singularity on ℜλ = 0 and is of slow increase as well as its derivatives.

For d_{−,ϕ,ϕ′} the proof is similar, but there is an extra step because of the presence of the intertwining operator. We have to assume that f_{1}^{1} is itself a convolution product f_{1}^{1} = f_{1}^{3} ∗ f_{1}^{4} and then write

J(ξ_{Q′}, …) = ∑_{ϕ′′} J(ξ_{Q′}, I_{wλ}(f_{1}^{3})ϕ′′, wλ_{0}^{Q})⟨M(w, λ)I_{λ}(f_{1}^{4})ϕ′, ϕ′′⟩.

The factor ⟨M(w, λ)I_{λ}(f_{1}^{4})ϕ′, ϕ′′⟩ has derivatives of slow increase so we can finish the proof as before.

It now follows that J_{χ}^{max}(f) is equal to

m(χ)v_{Q} ∑_{ϕ,ϕ′} ∫ [\frac{e^{⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}d_{+,ϕ,ϕ′}(λ) − \frac{e^{−⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩}d_{−,ϕ,ϕ′}(λ)] ⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩dλ

(note that v_{Q} = v_{Q′}). We shall compute the limit as T → ∞ by the method of the (G, M) families, which amounts here to writing the integrand as the sum of the following three terms:

e^{⟨λ_{Q},T ⟩} \frac{d_{+,ϕ,ϕ′}(λ) − d_{+,ϕ,ϕ′}(λ_{0}^{Q})}{⟨λ_{Q}, α̌_{Q}⟩}⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩,

−e^{−⟨λ_{Q},T ⟩} \frac{d_{−,ϕ,ϕ′}(λ) − d_{+,ϕ,ϕ′}(λ_{0}^{Q})}{⟨λ_{Q}, α̌_{Q}⟩}⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩,

(\frac{e^{⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩} − \frac{e^{−⟨λ_{Q},T ⟩}}{⟨λ_{Q}, α̌_{Q}⟩})d_{+,ϕ,ϕ′}(λ_{0}^{Q})⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩.

From the relation

d_{+,ϕ,ϕ′}(λ_{0}^{Q}) = d_{−,ϕ,ϕ′}(λ_{0}^{Q})

we see that in the two first terms the fractions represent smooth functions of λ which are of slow increase. Since ⟨I_{λ}(f_{1}^{2})ϕ, ϕ′⟩ is of rapid decrease, the product is integrable and the integrals of the first two terms tend to 0 as → ∞ by the Riemann–Lebesgue Lemma. In the third term, the expression in parentheses is the Fourier transform of the convex hull of −T_{Q}, T_{Q} where T_{Q} is the projection of T on 𝔞_{Q}. Hence its integral tends to

∫_{i𝔄_{0}^{Q}} d_{+,ϕ,ϕ′}(λ_{0}^{Q})⟨I_{λ_{0}^{Q}}(f_{1}^{2})ϕ, ϕ′⟩dλ_{0}^{Q}.

Thus we finally find

J_{χ}^{max}(f) = m(χ)v_{Q} ∑_{ϕ} ∫_{i𝔄_{0}^{Q∗}} J(ξ_{Q}, I_{λ_{0}^{Q}}(f_{1})ϕ, λ_{0}^{Q})\overline{W}_{λ}(I_{λ_{0}^{Q}}(f_{2})ϕ, λ_{0}^{Q})dλ_{0}^{Q}.

This formula has been established for a K-finite function f = f_{1}∗f_{2}^{∗} where f_{1} and f_{2} are K-finite and themselves convolution of K-finite functions. We will write λ for λ_{0}^{Q} and set s = ⟨λ, α̌_{1}⟩. We will also indicate the dependence on σ. We have seen that W_{λ,σ} and J(ξ_{Q}, ϕ, λ, σ) (which may not be defined at λ = 0) define continuous linear forms on the space of smooth vectors of the representation I_{λ,σ}. Denoting by W_{λ,σ} and J_{λ,σ} the corresponding generalized vectors, we can write the result in the form

J_{χ}^{max}(f) = m(χ)v_{Q} ∫ ⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩dλ.

The integrand is defined even at λ = 0. In fact, we can write the integrand in the form

⟨I_{λ,σ}(f)W_{λ,σ}^{1}, J_{λ,σ}^{1}⟩

×\frac{1}{|L^{S_{0}}(1 + 2s, σ_{1}|F ω_{E/F})|^{2}}

×\frac{1}{∏_{i<j,(i,j)≠(1,2)} L^{S}(1 + s_{j} − s_{i}, σ_{j}σ_{i}^{−1})}.

For our next step we let S_{0} be a finite set of places of F and S the corresponding set of places of E. As usual we assume that S_{0} contains all places at infinity. For v finite we fix a smooth function of compact support f_{v}. We assume it is the characteristic function of K_{v} for almost all v. For v infinite we let B_{v} be a bounded set of the space of smooth functions of compact support. Finally we let B be the set of functions f = ∏ f_{v} with f_{v} ∈ B_{v} for v infinite.

Lemma 17: Let B be the above set. Then there is a constant C > 0 such that, for f ∈ B,

∑_{σ} ∫ |⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩| dλ ≤ C,

the sum over all triples of the above type.

To prove the lemma, we may as well assume that f = f_{1} ∗ f_{2} ∗ f_{3} where the functions f_{i}, i = 1, 2, 3 remain in a set B. Using local majorizations, we see that there is a polynomial P (λ, σ) such that for ϕ of norm one and f ∈ B,

|J_{λ,σ}^{1}(I_{λ,σ}(f)ϕ)| ≤ |P (s, σ)|.

Otherwise,

∥ I_{λ,σ}(f)J_{λ,σ}^{1} ∥≤ |P (s, σ)|.

There is a similar estimate for W_{λ,σ}^{1}. We can write

⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩ = ⟨I_{λ,σ}(f_{2})I_{λ,σ}(f_{3})W_{λ,σ}^{1}, I_{λ,σ}(f_{1}^{∗})J_{λ,σ}^{1}⟩

and this is thus majorized by

∥ I_{λ,σ}(f_{2}) ∥ |P^{1}(s, σ)|,

where P^{1} is a polynomial. Using estimates for the L-factors, we are reduced to estimate an expression of the form

∑_{σ} ∫ ∥ I_{λ,σ}(f) ∥ |P (s, σ)| dλ,

for f ∈ B. Now I_{λ,σ}(f) is the operator defined by the kernel

L(k_{1}, k_{2}) = ∫ f(k_{1}^{−1}ank_{2})dne^{⟨λ+ρ,H(a)⟩}dnda.

Thus

∥ I_{λ,σ}(f_{2}) ∥≤ sup_{k_{1},k_{2}} |L(k_{1}, k_{2})| .

For k_{i} ∈ K, f ∈ B the functions

a ↦ ∫ f(k_{1}^{−1}ank_{2})dne^{⟨ρ,H(a)⟩}

remain in a bounded set B_{A} of the space of smooth functions of compact support on A(𝔸). In particular, they are invariant under A(E_{𝔸}) ∩ K^{S}. We have then

∥ I_{λ,σ}(f_{2}) ∥≤ sup_{ϕ∈B_{A}} |∫ ϕ(a)σ(a)e^{⟨λ,H(a)⟩}da| .

Finally, given P (λ, σ) and a bounded set B_{A}, we need to show that there is a constant C such that

∑_{σ} ∫ |∫ ϕ(a)σ(a)e^{⟨λ,H(a)⟩}da| |P (λ, σ)|dλ ≤ C,

for ϕ ∈ B_{A}. Our assertion follows.

Now we finish the proof of Proposition 10. By the previous lemma,

f ↦ m(χ)v_{Q} ∫ ⟨I_{λ,σ}(f)W_{λ,σ}, J_{λ,σ}⟩dλ

is a distribution. It is equal to J_{χ}(f) when f is K-finite and a convolution of sufficiently many K-finite functions. Thus it is in fact equal to J_{χ}(f) for all f.

9. The discrete part of the trace formula

Now we claim that, for any smooth function of compact support,

(59)  J(f) = ∑_{discrete} J_{χ}(f) + ∑_{μ_{1}} J_{μ_{1}}(f).

In the first sum, χ appears if it is represented by a pair (M, π) where π is a distinguished representation of M and, furthermore, n(χ) = n_{1}(χ). The restriction of the central character of π to center of G must be equal to ω. Then

J_{χ}(f) = ⟨I_{π}(f)W_{π}, J_{π}⟩,

where W_{π} is the generalized vector corresponding to a Whittaker linear form and J_{π} the generalized vector corresponding to the period integral

J_{π}(ϕ) = ∫_{K_{H}} ∫_{M_{H}} ϕ(hk)dhdk.

Explicitly, the possibilities are M = G, M is of type (2, 1) and M = A, with π = (σ_{1}, σ_{2}, σ_{3}), the three characters being distinct and distinguished. The second sum is over all idele class characters μ_{1} of E. Then

J_{μ_{1}}(f) = \frac{1}{2}v_{Q} ∫ ⟨I_{λ,π}(f)W_{λ,π}, J_{λ,π}⟩dλ

where π is the representation induced by (μ_{1}, \overline{μ}_{1}^{−1}, μ_{2}), μ_{2} is the distinguished idele class character of E such that \overline{μ}_{1}^{−1}μ_{2} = ω, and λ is integrated over the dual of i𝔞_{0}^{Q}. Furthermore, the expression is absolutely convergent. More precisely, if B is a bounded set in the space of smooth functions of compact support on G(E_{𝔸}) then there is C > 0 such that, for f ∈ B,

∑_{discrete} |J_{χ}(f)| + ∑ \frac{1}{2}v_{Q} ∫ |⟨I_{λ,π}(f)W_{λ,π}, J_{λ,π}⟩| dλ < C.

To prove the equality (59), we remark that we have established it when f is the convolution of sufficiently many K-finite functions. Since both sides are distributions, they must be equal for all functions f.

10. Convergence for the unitary group

We go back to the notations of the introduction. Thus U is the unitary group for the matrix w. If μ is an idele class character of E, we consider the space I(μ, ζ) of smooth functions ϕ: U(F_{𝔸}) → ℂ such that

ϕ(ud_{a}ng) = ϕ(g)μ(a)ζ(u).

We set

e^{⟨λ+ρ,H(g)⟩} =| a |^{1+s}

if g = n′d_{a}uk, with k in the standard maximal compact subgroup K′. For ϕ a K′-finite function in I(μ, σ), we define the corresponding Eisenstein series

E(g, ϕ; λ, μ, ζ) = ∑ ϕ(γg)e^{⟨λ+ρ,H(γg)⟩}.

We define a linear form W_{λ,μ,ζ} on the space of K-finite vectors of I(μ, ζ) by

W_{λ,μ,ζ}(f) = ∫ E(n, ϕ; λ, μ, ζ)θ(n)dn.

This is the analytic continuation of the integral

∫ ϕ(wn)e^{⟨λ+ρ,H(wn)⟩}θ(n)dn.

Again, the linear form extends to the space of smooth vectors. In fact, if S_{0} and S are as before, then the restriction of the linear form to the space of vectors invariant under K^{S_{0}} is given by

W_{λ,μ,ζ}(ϕ) = W_{λ,μ,ζ}^{1}(ϕ) × \frac{1}{L^{S}(s + 1, μ)L^{S_{0}}(s + 1, ω_{E/F} μ|F)},

where W_{λ,μ,ζ}^{1} is defined locally and holomorphic in λ. Let us now consider a datum χ. Then the kernel associated to a K-finite function is given by

K_{χ}(x, y) = ∫ ∑_{ϕ} E(x, I_{λ,μ,χ}(f)ϕ; λ)\overline{E(y, ϕ; λ)}dλ + ⋯,

where the terms ⋯ correspond to residues of the Eisenstein series. For the purpose of showing that the trace formula gives an absolutely convergent expression, it is more convenient to define

J_{χ}(f) = ∫ K_{χ}(n_{1}, n_{2})θ(n_{1})\overline{θ(n_{2})}dn_{1}dn_{2}.

Because the linear form W_{λ,μ,ζ} is holomorphic for ℜs ≥ 0 and the residual spectrum is obtained by taking the residue at appropriate points for ℜs > 0, we get

J_{χ}(f) = ∫ ∑_{ϕ} W_{λ,μ,ζ}(I_{λ,μ,ζ}(f)ϕ)\overline{W_{λ,μ,ζ}(ϕ)}dλ.

We can introduce a generalized vector W_{λ,μ,ζ} and write the above expression as

J_{χ}(f) = ∫ ⟨I_{λ,μ,ζ}(f)W_{λ,μ,ζ}, W_{λ,μ,ζ}⟩dλ.

We want to prove the following: if f is a smooth function of compact support, then

∑_{μ} ∫ |⟨I_{λ,μ,ζ}(f)W_{λ,μ,ζ}, W_{λ,μ,ζ}⟩| dλ < +∞.

We may assume that f is a convolution product and even that

f = f_{1}^{∗} ∗ f_{1},

where f_{1} is a smooth function of compact support. Then the expression reads

∑_{μ,ζ} ∫ ∥I_{λ,μ,ζ}(f_{1})W_{λ,μ,ζ}∥^{2}dλ.

If f_{1} is K-finite, this is indeed finite. In fact, there is a continuous semi-norm m on the space of smooth functions of compact support such that the above expression is bounded by m(f_{1}^{∗} ∗ f_{1}) for f_{1} K-finite. The general case can be obtained in the following way: given f_{1} we may write f_{1} = lim f_{α} where (f_{α}) is a sequence of smooth K-finite functions of compact support. The functions f_{1} and (f_{α}) have support in a fixed compact set and f_{α} converges uniformly to f and any invariant derivative of f_{α} converges uniformly to the corresponding derivative of f_{1}. We have

∑_{μ} ∫ ∥I_{λ,μ,ζ}(f_{α} − f_{β})W_{λ,μ,ζ}∥^{2}dλ ≤ m((f_{α} − f_{β})(f_{α} − f_{β})^{∗}).

As α and β tend to infinity the right hand side tends to 0. We may regard all the Hilbert spaces I_{μ} as closed subspaces of the same Hilbert space H := L^{2}(K, dk). We may also regard X = i𝔄_{0}^{∗}×{ \atop E_{𝔸}^{×}\E^{×}} as a measure space (X, dx). The functions

(λ, μζ) ↦ I_{λ,μ,ζ}(f)W_{λ,μ,ζ}

are then measurable functions on X with values in H. Then the sequence of functions

A_{α}: (λ, μ) ↦ I_{λ,μ,ζ}(f_{α})W_{λ,μ,ζ}

is a Cauchy sequence in the Hilbert space L^{2}(X, dx, H). It converges pointwise to the function

A_{1}: (λ, μ) ↦ I_{λ,μ,ζ}(f_{1})W_{λ,μ,ζ}.

Thus a subsequence of A_{α} converges pointwise and in the Hilbert space to the function A_{1}. It follows that the function A_{1} is in the same Hilbert space. Moreover,

∑_{μ} ∫ ∥I_{λ,μ,ζ}(f_{1})W_{λ,μ,ζ}∥^{2}dλ ≤ m(f_{1}^{∗} ∗ f_{1}).

11. Conclusion We are now ready to prove the main result of this paper.

Theorem 2: Any stable tempered packet of cuspidal representations of U contains a globally generic representation.

Proof: Indeed the packet has for base change a single irreducible cuspidal representation Π of GL(3, E_{𝔸}). By the results discussed in the second section the representation Π is distinguished. We go back to the discussion of the introduction section. If f and f′ are as in the introduction, the absolute convergence of the relative trace formula allows us to write

J_{Π}(f) = ∑_{π} J′_{π}(f′)

where the sum is over all the members of the packet. Since the distribution J_{Π}(f) is a product of local relative Bessel distributions, we can choose a function f satisfying our simplifying assumption such that the left hand side is non-zero. It follows that the right hand side is non-zero and there is at least one π such that J′_{π}(f′) ≠ 0. Such a π is generic.

The same proof applies to the case of an endoscopic packet, by considering the discrete, non-cuspidal, part of the trace formula. At any rate, the result is already known as a consequence of the theory of the Weil representation ([GRS]). It is also a consequence of another trace formula (cf. [F4], and also [Mao2]).

We can also prove a local result.

Theorem 3: Let E/F be a local quadratic extension. Any tempered L-packet of U(F) contains exactly one generic component.

Proof: By [GRS2], the only case which is not known is the case of a local packet reduced to a single supercuspidal representation π. Let us write the local quadratic extension as E_{v}/F_{v_{0}} where E/F is a quadratic extension of number fields. Let us write π_{v_{0}} for π. There is an irreducible cuspidal representation π of U(F_{𝔸}) having the component π_{v_{0}}. By the previous theorem there is a cuspidal generic representation π′ in the same L-packet as π. In particular π′_{v_{0}} = π_{v_{0}}, and the theorem follows.

We have also the following corollary.

Theorem 4: Suppose that π is a cuspidal automorphic representation of U(F_{𝔸}) such that π_{v_{0}} is generic for every place v_{0}. Then π is globally generic.

Proof: Indeed, there is a cuspidal automorphic π′ in the same packet as π. By the previous result π_{v_{0}} = π′_{v_{0}} for all v_{0}. Thus π = π′.

References

[A1] J. Arthur, A trace formula for reductive groups I: Terms associated to classes in G(ℚ), Duke Mathematical Journal 45 (1978), 912–952.

[A2] J. Arthur, A trace formula for reductive groups II: Applications of a truncation operator, Compositio Mathematica 40 (1980), 87–121.

[A3] J. Arthur, The trace formula in invariant form, Annals of Mathematics 114 (1981), 1–74.

[As] T. Asai, On Certain Dirichlet series associated with Hilbert modular forms, Mathematische Annalen 226 (1977), 81–94.

[Ba] E. M. Baruch, Proof of Kirillov’s conjecture, preprint.

[BM] E. M. Baruch and Z. Mao, Central critical value of automorphic L-functions, preprint

[B] J. Bernstein, P-invariant distributions of GL(n) and the classification of unitary representations of GL(n) (non-Archimedean case), in Lie Group Representations II, College Park, Md., 1982/1983, Lecture Notes in Mathematics 1041, Springer-Verlag, Berlin, 1984.

[CHM] W. Casselman, H. Hecht and D. Miličié, Bruhat filtrations and Whittaker vectors for real groups, preprint.

[CJ] N. Chen and H. Jacquet, Positivity of quadratic base change L functions, to appear in Bulletin de la Société Mathématique de France

[CS] W. Casselman and F. Shahidi, On irreducibility of standard modules for generic representations, Annales Scientifiques de l’École Normale Supérieure 31 (1998), 561–589.

[DM] J. Dixmier and P. Malliavin, Factorizations de fonctions et vecteurs indéfiniment différentiables, Bulletin de la Société Mathématique de France 102 (1978), 305–330.

[F1] Y. Flicker, Twisted tensors and Euler products, Bulletin de la Société Mathématique de France 116 (1988), 295–313.

[F2] Y. Flicker, On distinguished representations, Journal für die reine und angewandte Mathematik 418 (1991), 139–172.

[F3] Y. Flicker, Distinguished Representations and a Fourier Trace formula, Bulletin de la Société Mathématique de France 120 (1992), 413–465.

[F4] Y. Flicker, Cyclic automorphic forms on a unitary group, Journal of Mathematics of Kyoto University 37 (1997), 347–439.

[F5] Y. Flicker, Automorphic forms with anisotropic periods on a unitary group, Journal of Algebra 220 (1999), 636–663.

[FZ] Y. Flicker and D. Zinoniev, On the poles of twisted L-functions, Proceedings of the Japan Academy of Science 71 (1995), 114–116.

[FGJR] F. Friedberg, S. Gelbart, H. Jacquet and J. Rogawski, Représentations génériques du groupe unitaire à trois variables, Comptes Rendus de l’Académie des Sciences, Paris, Série I 329 (1999), 255–260.

[GRS1] S. Gelbart, J. Rogawski and D. Soudry, Endoscopy of cusp forms and algebraic cycles for U(3), Israel Journal of Mathematics 83 (1993), 213–252.

[GRS2] S. Gelbart, J. Rogawski and D. Soudry, Endoscopy, theta liftings and period integrals for the unitary group in three variables, Annals of Mathematics 145 (1997), 419–476.

[GK] I. M. Gelfand and D. Kazhdan, Representations of GL(n, K) where K is a local field, in Lie Groups and their Representations (Proc. Summer School, Bolyai Janos Math. Soc., Budapest, 1971), Halsted, New York, 1975, pp. 95–118.

[Gol] D. Goldberg, Some results on reducibility for unitary groups and local Asai L-functions, Journal für die reine und angewandte Mathematik 448 (1994), 65–95.

[HLR] G. Harder, R. P. Langlands and M. Rapoport, Algebraische Zyklen auf Hilbert–Blumenthal-Flächen (German) [Algebraic cycles on Hilbert– Blumenthal surfaces], Journal für die reine und angewandte Mathematik 366 (1986), 53–120.

[J1] H. Jacquet, Generic representations, in Non-Commutative Harmonic Analysis, Actes Colloq., Marseille-Luminy, Lectures Notes in Mathematics 587, Springer-Verlag, Berlin, 1977.

[J2] H. Jacquet, Relative Kloosterman Integrals for GL(3): II, Canadian Journal of Mathematics 44 (1987), 117–155.

[J3] H. Jacquet, The continuous spectrum of the relative trace formula for GL(3) over a quadratic extension, Israel Journal of Mathematics 89 (1995), 1–59.

[JLR] H. Jacquet, E. M. Lapid and J. Rogawski, Periods of automorphic forms, Journal of the American Mathematical Society 12 (1999), 173–240.

[JS] H. Jacquet and J. Shalika, On Euler products and the classification of automorphic representations I, II, American Journal of Mathematics 103 (1981), 449–558 and 777–815.

[KRS] S. Kudla, S. Rallis and S. Soudry, On the degree 5 L-function for Sp(2), Inventiones Mathematicae 107 (1992), 483–541.

[KV] J. Kolk and V. S. Varadarajan, On the transverse symbol of vectorial distributions and some applications to harmonic analysis, Indagationes Mathematicae (N.S.) 7 (1996), 67–96.

[L] S. Lang, Algebraic Number Theory, Springer-Verlag, Berlin, 1986.

[Mao1] Mao, Z., Relative Kloosterman integrals for the unitary group, Comptes Rendus de l’Académie des Sciences, Paris, Série I 315 (1992), 381–386.

[Mao2] Z. Mao, Relative Kloosterman integrals for GL(3): III, Canadian Journal of Mathematics 45 (1992), 1211–1230.

[MW] C. Moeglin and J. L. Waldspurger, Le spectre résiduel de GL(n), Annales Scientifiques de l’École Normale Supérieure 22 (1989), 605–704.

[Ro1] J. Rogawski, Automorphic Representations of Unitary Groups in Three Variables, Annals of Mathematics Studies, No. 123, Princeton University Press, 1990.

[Ro2] J. Rogawski, The multiplicity Theorem for A-packets, in The Zeta Functions of Picard Modular Surfaces, Les publications CRM, Montreal, 1992.

[fS1] F. Shahidi, Automorphic L-functions: a survey, in Automorphic Forms, Shimura Varieties, and L-functions, Vol. I, Ann Arbor, MI, 1988, pp. 415– 437.

[fS2] F. Shahidi, A proof of Langlands’ conjecture on Plancherel measure; complementary series for p-adic groups, Annals of Mathematics 132 (1990), 273–330.

[fS3] F. Shahidi, On the Ramanujan conjecture and finiteness of poles for certain L-functions, Annals of Mathematics 127 (1988), 547–584.

[jS] J. Shalika, The multiplicity one theorem for GL_{n}, Annals of Mathematics 100 (1974), 171–193.

[W] N. Wallach, Real Reductive Groups II, Pure and Applied Mathematics, Vol. 132-II, Academic Press, Boston, 1992.

[Y] Y. Ye, Kloosterman integrals and base change, Journal für die reine und angewandte Mathematik 400 (1989), 57–121.