FOURIER NEURAL OPERATOR FOR PARAMETRIC PARTIAL DIFFERENTIAL EQUATIONS

Peng Jian

1.Abstract

Neural operators that learn mappings between function spaces
formulate a new neural operator by parameterizing the integral kernel directly in Fourier space(频率空间)
$f:R\to R,\mathcal{F}:f\to g,g:k(频率)\to C$

2.Introduction

Two mainstream neural network-based approaches for PDEs – the finite-dimensional operators and Neural-FEM
The integral operator is restricted to a convolution, and instantiated through a linear transformation in the Fourier domain
The proposed framework can approximate complex operators raising in PDEs that are highly non-linear, with high frequency modes and slow energy decay
优点：
They learn an entire family of PDEs, in contrast to classical methods which solve on instance of the equation
Three orders of magnitude faster compared to traditional PDE solvers.
Achieves superior accuracy compared to previous learning-based solvers under fixed resolution

3.Learning Operators

3.1符号说明

有界定义域：$D\subset {\mathbb{R}}^d$

separable Banach spaces of function：$\mathcal{A}=\mathcal{A}(D;{\mathbb{R}}^{d_a})$ and $\mathcal{U}=\mathcal{U}(D;{\mathbb{R}}^{d_u})$

函数对集合：{aj,uj}j=1N where aj∼μ is an i.i.d. sequence\{a_j,u_j\}_{j=1}^N\text{ where }a_j\sim\mu\text{ is an i.i.d. sequence}{aj​,uj​}j=1N​ where aj​∼μ is an i.i.d. sequence

learns a mapping between two infinite dimensional spaces from a finite collection of observed input-output pairs

non-linear map：$G^{†}:\mathcal{A}\to \mathcal{U}$，$\begin{array}{r}{u}_j=G^{†}(a_j)\end{array}$

We aim to build an approximation of $G^{†}$ by constructing a parametric map(参数映射)

parametric map：$G:\mathcal{A}\times \Theta \to \mathcal{U}\text{or equivalently},G_{\theta }:\mathcal{A}\to \mathcal{U},\theta \in \Theta$

cost function：$C:\mathcal{U}\times \mathcal{U}\to \mathbb{R}$

seek a minimizer of the problem：${\min }_{\theta \in \Theta }{\mathbb{E}}_{a\sim \mu }[C(G(a,\theta ),G^{†}(a))]$

3.2Discretization(离散化)

Since our data $a_j,u_j$ are, in general, functions, to work with them numerically,we assume access only to point-wise evaluations(逐点评估).

Dj = {x1,…,xn} ⊂ D D_j~=~\{x_1,\ldots,x_n\}~\subset~D~Dj​ = {x1​,…,xn​} ⊂ D  be a $n\text{-}$point discretization of the domain $D$

$\begin{array}{r}{a}_j{|}_{D_j}\in {\mathbb{R}}^{n\times d_a},u_j{|}_{D_j}\in {\mathbb{R}}^{n\times d_v}\end{array}$

To be discretization-invariant, the neural operator can produce an answer $u(x)$ for any $x\in D$, potentially $x\notin D_j$

4.Neural operator

神经算子的本质：Iterative architecture

$v_0\mapsto v_1\mapsto \dots \mapsto v_T$

where $\begin{array}{r}{v}_j\text{ for }j=0,1,\dots ,T-1\end{array}$ is a sequence of functions each taking values in ${\mathbb{R}}^{d_{\upsilon }}$

higher dimensional representation of $a(x)$：$v_0(x)=P(a(x))$, where $P$ is a fully connected neural network

output：$\begin{array}{r}u(x)=Q(v_T(x))\end{array}$, where $Q:{\mathbb{R}}^{d_v}\to {\mathbb{R}}^{d_u}$

神经算子的强大之处在于将线性的全局积分算子（通过傅里叶变换）与非线性的局部激活函数相结合。类似于标准神经网络通过将线性乘法与非线性激活函数相结合来逼近高度非线性函数

迭代关系$\begin{array}{r}{v}_t\mapsto v_{t+1}\end{array}$：

$v_{t+1}(x)=\sigma (W{v}_t(x)+{\int }_D\kappa (x,y,a(x),a(y);\varphi )v_t(y)\mathrm{d}y)$

因而可以理解下图：

5.Fourier Neural operator

进一步改进：用卷积算子(傅立叶变换)替换神经算子中的核积分算子，相应地，定义域发生改变(核积分算子自变量为$x$，卷积算子自变量为$k$ (频率))

Fourier Transform：

$(\mathcal{F}f{)}_j(k)={\int }_D{f}_j(x)e^{-2i\pi ⟨x,k⟩}\mathrm{d}x$

inverse：

$({\mathcal{F}}^{-1}f{)}_j(x)={\int }_D{f}_j(k)e^{2i\pi ⟨x,k⟩}\mathrm{d}k$

改进操作：令核积分算子中的 ${\kappa }_{\varphi }(x,y,a(x),a(y))={\kappa }_{\varphi }(x-y)$

那么$\left({\mathcal{K}}_{\varphi }{v}_t\right)(x)=({\mathcal{K}}_{\varphi }\ast v_t)(x)$

根据卷积定理：$\mathcal{F}({\mathcal{K}}_{\varphi }\ast v_t)=\mathcal{F}({\mathcal{K}}_{\varphi })\ast \mathcal{F}(v_t)$

所以：${\mathcal{K}}_{\varphi }\ast v_t={\mathcal{F}}^{-1}\mathcal{F}({\mathcal{K}}_{\varphi }\ast v_t)={\mathcal{F}}^{-1}(\mathcal{F}({\mathcal{K}}_{\varphi })\mathcal{F}(v_t))$

最终得：$\left(\mathcal{K}(a;\varphi )v_t\right)(x)={\mathcal{F}}^{-1}\left(\mathcal{F}({\kappa }_{\varphi })\cdot \mathcal{F}(v_t)\right)(x),\forall x\in D.$

directly parameterize ${\kappa }_{\varphi }$ in Fourier space then let $R_{\varphi }=\mathcal{F}({\kappa }_{\varphi })$

因此：$v_{t+1}(x)=\sigma (W{v}_t(x)+{\mathcal{F}}^{-1}\left(R_{\varphi }\cdot (\mathcal{F}{v}_t)\right)(x))$

卷积算子自变量为$k$ (频率模式 frequency mode)

所以：$(\mathcal{F}{v}_t)(k)\in {\mathbb{C}}^{d_v}\mathrm{and}{R}_{\varphi }(k)\in {\mathbb{C}}^{d_v\times d_v}$ (参考Abstract)

前文已经假设${\kappa }_{\varphi }$是周期函数，可以展开成傅立叶级数形式，so we may work with the discrete modes $k\in {\mathbb{Z}}^d$

截断傅立叶级数：

We pick a fifinite-dimensional parameterization by truncating(截断) the Fourier series at a maximal number of modes(最大模式数)kmax⁡=∣Zkmax⁡∣=.∣{k∈Zd:∣kj∣≤kmax⁡,j, for j=1,…,d}∣,Zkmax⁡=.{k∈Zd:∣kj∣≤kmax⁡,j, for j=1,…,d}k_{\max}=|Z_{k_{\max}}|\stackrel{.}{=}|\{k\in\mathbb{Z}^d:|k_j|{\leq}k_{\max,j},\text{ for }j=1,\ldots,d\}|,\quad Z_{k_{\max}}\stackrel{.}{=}\{k\in\mathbb{Z}^d:|k_j|{\leq}k_{\max,j},\text{ for }j=1,\ldots,d\}kmax​=∣Zkmax​​∣=.∣{k∈Zd:∣kj​∣≤kmax,j​, for j=1,…,d}∣,Zkmax​​=.{k∈Zd:∣kj​∣≤kmax,j​, for j=1,…,d}

因为$R_{\varphi }(k)\in {\mathbb{C}}^{d_v\times d_v}$，所以$R_{\varphi }(Z_{k_{\max }})\in {\mathbb{C}}^{k_{\max }\times d_v\times d_v}$， 所以直接参数化$R_{\varphi }$为一组复值张量(complex-valued$(k_{\max }\times d_v\times d_v)$ -tensor)，因此可以省略符号$\varphi$

总结：周期函数–>傅立叶展开–>使用离散的频率模式表示不同的频率分量–>设定最大模式数–>截断傅立叶级数–>得到周期函数的近似–>进而得到$R_{\varphi }$–>得到复值张量(参数化表示)

共轭对称性(conjugate symmetry)：由于函数 $\kappa$ 是实值的，对应的傅里叶变换结果 $R_{\varphi }$ 具有共轭对称性。这意味着在频域中的负频率模式的振幅和相位与正频率模式相同，只是共轭的。在实际应用中，利用共轭对称性可以减少计算量。由于正负频率模式具有相同的振幅和相位（只是共轭关系），我们只需要计算正频率模式的傅里叶系数，并根据共轭对称性得到负频率模式的系数。

5.1The discrete case and the FFT.

假设区域$D$离散化为$n$个点，那么$v_t\in {\mathbb{R}}^{n\times d_v}\text{ and }\mathcal{F}(v_t)\in {\mathbb{C}}^{n\times d_v}$，因为我们将$v_t$和只有$k_{\max }$个傅立叶模式的函数进行卷积，因此我们可以进行截断，得到$\mathcal{F}(v_t)\in {\mathbb{C}}^{k_{\max }\times d_v}$。

所以${\left(R\cdot (\mathcal{F}{v}_t)\right)}_{k,l}={\sum }_{j=1}^{d_v}{R}_{k,l,j}(\mathcal{F}{v}_t{)}_{k,j},k=1,\dots ,k_{\max },j=1,\dots ,d_v$，得到$k_{\max }\times d_v$的矩阵，每一行都是$v_{t+1}(x)=\sigma (W{v}_t(x)+(\mathcal{K}(a;\varphi )v_t)(x))$中激活函数的一个输入。

When the discretization is uniform with resolution $s_1\times \cdots \times s_d=n,d=\dim (D)$, $\mathcal{F}$ can be replaced by the Fast Fourier Transform

快速傅立叶变换：

$f\in {\mathbb{R}}^{n\times d_v},k=(k_1,\dots ,k_d)\in {\mathbb{Z}}_{s_1}\times \cdots \times {\mathbb{Z}}_{s_d},\text{and }x=(x_1,\dots ,x_d)\in D$

快速傅立叶变换$FFT$ $\hat{\mathcal{F}}$

逆变换：${\hat{\mathcal{F}}}^{-1}$

$\begin{array}{rl}(\hat{\mathcal{F}}f{)}_l(k)& =\sum _{x_1=0}^{s_1-1}\cdots \sum _{x_d=0}^{s_d-1}{f}_l(x_1,\dots ,x_d)e^{-2i\pi {\sum }_{j=1}^d\frac{x_j{k}_j}{s_j}},\\ ({\hat{\mathcal{F}}}^{-1}f{)}_l(x)& =\sum _{k_1=0}^{s_1-1}\cdots \sum _{k_d=0}^{s_d-1}{f}_l(k_1,\dots ,k_d)e^{2i\pi {\sum }_{j=1}^d\frac{x_j{k}_j}{s_j}}\end{array}$

$R$ is treated as a $(s_1\times \cdots \times s_d\times d_v\times d_v)$-tensor