Molecular modelling

In mathematics and computer algebra, automatic differentiation (AD), also called algorithmic differentiation or computational differentiation, is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

Automatic differentiation is not:

Figure 1: How automatic differentiation relates to symbolic differentiation

• Symbolic differentiation, or
• Numerical differentiation (the method of finite differences).

These classical methods run into problems: symbolic differentiation leads to inefficient code (unless carefully done) and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where the complexity and errors increase. Finally, both classical methods are slow at computing the partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems.

The chain rule, forward and reverse accumulation

Fundamental to AD is the decomposition of differentials provided by the chain rule. For the simple composition $f(x)=g(h(x))$ the chain rule gives

${\frac {df}{dx}}={\frac {dg}{dh}}{\frac {dh}{dx}}$

Usually, two distinct modes of AD are presented, forward accumulation (or forward mode) and reverse accumulation (or reverse mode). Forward accumulation specifies that one traverses the chain rule from right to left (that is, first one computes $dh/dx$ and then $dg/dh$), while reverse accumulation has the traversal from left to right.

Figure 2: Example of forward accumulation with computational graph

Forward accumulation

Forward accumulation automatic differentiation is the easiest to understand and to implement. The function $f(x_{1},x_{2})=x_{1}x_{2}+\sin(x_{1})$ is interpreted (by a computer or human programmer) as the sequence of elementary operations on the work variables $w_{i}$, and an AD tool for forward accumulation adds the corresponding operations on the second component of the augmented arithmetic.

Original code statements	Added statements for derivatives
$w_{1}=x_{1}$	$w'_{1}=1$ (seed)
$w_{2}=x_{2}$	$w'_{2}=0$ (seed)
$w_{3}=w_{1}w_{2}$	$w'_{3}=w'_{1}w_{2}+w_{1}w'_{2}=1x_{2}+x_{1}0=x_{2}$
$w_{4}=\sin(w_{1})$	$w'_{4}=\cos(w_{1})w'_{1}=\cos(x_{1})1$
$w_{5}=w_{3}+w_{4}$	$w'_{5}=w'_{3}+w'_{4}=x_{2}+\cos(x_{1})$

The derivative computation for $f(x_{1},x_{2})=x_{1}x_{2}+\sin(x_{1})$ needs to be seeded in order to distinguish between the derivative with respect to $x_{1}$ or $x_{2}$. The table above seeds the computation with $w'_{1}=1$ and $w'_{2}=0$ and we see that this results in $x_{2}+\cos(x_{1})$ which is the derivative with respect to $x_{1}$. Note that although the table displays the symbolic derivative, in the computer it is always the evaluated (numeric) value that is stored. Figure 2 represents the above statements in a computational graph.

In order to compute the gradient of this example function, that is $\partial f/\partial x_{1}$ and $\partial f/\partial x_{2}$, two sweeps over the computational graph is needed, first with the seeds $w'_{1}=1$ and $w'_{2}=0$, then with $w'_{1}=0$ and $w'_{2}=1$.

The computational complexity of one sweep of forward accumulation is proportional to the complexity of the original code.

Forward accumulation is superior to reverse accumulation for functions $f:\mathbb {R} \rightarrow \mathbb {R} ^{m}$ with $m\gg 1$ as only one sweep is necessary, compared to $m$ sweeps for reverse accumulation.

Figure 3: Example of reverse accumulation with computational graph

Reverse accumulation

Reverse accumulation traverses the chain rule from left to right, or in the case of the computational graph in Figure 3, from top to bottom. The example function is real-valued, and thus there is only one seed for the derivative computation, and only one sweep of the computational graph is needed in order to calculate the (two-component) gradient. This is only half the work when compared to forward accumulation, but reverse accumulation requires the storage of some of the work variables $w_{i}$, which may represent a significant memory issue.

The data flow graph of a computation can be manipulated to calculate the gradient of its original calculation. This is done by adding an adjoint node for each primal node, connected by adjoint edges which parallel the primal edges but flow in the opposite direction. The nodes in the adjoint graph represent multiplication by the derivatives of the functions calculated by the nodes in the primal. For instance, addition in the primal causes fanout in the adjoint; fanout in the primal causes addition in the adjoint; a unary function $y=f(x)$ in the primal causes $x'=f'(x)y'$ in the adjoint; etc.

Reverse accumulation is superior to forward accumulation for functions $f:\mathbb {R} ^{n}\rightarrow \mathbb {R}$ with $n\gg 1$, where forward accumulation requires roughly n times as much work.

Backpropagation of errors in multilayer perceptrons, a technique used in machine learning, is a special case of reverse mode AD.

Jacobian computation

The Jacobian $J$ of $f:\mathbb {R} ^{n}\rightarrow \mathbb {R} ^{m}$ is an $m\times n$ matrix. The Jacobian can be computed using $n$ sweeps of forward accumulation, of which each sweep can yield a column vector of the Jacobian, or with $m$ sweeps of reverse accumulation, of which each sweep can yield a row vector of the Jacobian.

Beyond forward and reverse accumulation

Forward and reverse accumulation are just two (extreme) ways of traversing the chain rule. The problem of computing a full Jacobian of $F:\mathbb {R} ^{n}\rightarrow \mathbb {R} ^{m}$ with a minimum number of arithmetic operations is known as the "optimal Jacobian accumulation" (OJA) problem. OJA is NP-complete. Central to this proof is the idea that there may exist algebraic dependences between the local partials that label the edges of the graph. In particular, two or more edge labels may be recognized as equal. The complexity of the problem is still open if it is assumed that all edge labels are unique and algebraically independent.

Automatic differentiation using dual numbers

Forward mode automatic differentiation is accomplished by augmenting the algebra of real numbers and obtaining a new arithmetic. An additional component is added to every number which will represent the derivative of a function at the number, and all arithmetic operators are extended for the augmented algebra. The augmented algebra is the algebra of dual numbers. Computer programs often implement this using the complex number representation.

Replace every number $\,x$ with the number $x+x'\varepsilon$, where $x'$ is a real number, but $\varepsilon$ is nothing but a symbol with the property $\varepsilon ^{2}=0$. Using only this, we get for the regular arithmetic

$(x+x'\varepsilon )+(y+y'\varepsilon )=x+y+(x'+y')\varepsilon$

$(x+x'\varepsilon )\cdot (y+y'\varepsilon )=xy+xy'\varepsilon +yx'\varepsilon +x'y'\varepsilon ^{2}=xy+(xy'+yx')\varepsilon$

and likewise for subtraction and division.

Now, we may calculate polynomials in this augmented arithmetic. If $P(x)=p_{0}+p_{1}x+p_{2}x^{2}+\cdots +p_{n}x^{n}$, then

$P(x+x'\varepsilon )$	$=\,$	$p_{0}+p_{1}(x+x'\varepsilon )+\cdots +p_{n}(x+x'\varepsilon )^{n}$
	$=\,$	$p_{0}+p_{1}x+\cdots +p_{n}x^{n}$
		$\,{}+p_{1}x'\varepsilon +2p_{2}xx'\varepsilon +\cdots +np_{n}x^{n-1}x'\varepsilon$
	$=\,$	$P(x)+P^{(1)}(x)x'\varepsilon$

where $P^{(1)}$ denotes the derivative of $P$ with respect to its first argument, and $x'$, called a seed, can be chosen arbitrarily.

The new arithmetic consists of ordered pairs, elements written $\langle x,x'\rangle$, with ordinary arithmetics on the first component, and first order differentiation arithmetic on the second component, as described above. Extending the above results on polynomials to analytic functions we obtain a list of the basic arithmetic and some standard functions for the new arithmetic:

$\langle u,u'\rangle +\langle v,v'\rangle =\langle u+v,u'+v'\rangle$

$\langle u,u'\rangle -\langle v,v'\rangle =\langle u-v,u'-v'\rangle$

$\langle u,u'\rangle *\langle v,v'\rangle =\langle uv,u'v+uv'\rangle$

$\langle u,u'\rangle /\langle v,v'\rangle =\left\langle {\frac {u}{v}},{\frac {u'v-uv'}{v^{2}}}\right\rangle \quad (v\neq 0)$

$\sin \langle u,u'\rangle =\langle \sin(u),u'\cos(u)\rangle$

$\cos \langle u,u'\rangle =\langle \cos(u),-u'\sin(u)\rangle$

$\exp \langle u,u'\rangle =\langle \exp u,u'\exp u\rangle$

$\log \langle u,u'\rangle =\langle \log(u),u'/u\rangle \quad (u>0)$

$\langle u,u'\rangle ^{k}=\langle u^{k},ku^{k-1}u'\rangle \quad (u\neq 0)$

$\left|\langle u,u'\rangle \right|=\langle \left|u\right|,u'{\mbox{sign}}u\rangle \quad (u\neq 0)$

and in general for the primitive function $g$,

$g(\langle u,u'\rangle ,\langle v,v'\rangle )=\langle g(u,v),g_{u}(u,v)u'+g_{v}(u,v)v'\rangle$

where $g_{u}$ and $g_{v}$ are the derivatives of $g$ with respect to its first and second arguments, respectively.

When a binary basic arithmetic operation is applied to mixed arguments—the pair $\langle u,u'\rangle$ and the real number $c$—the real number is first lifted to $\langle c,0\rangle$. The derivative of a function $f:\mathbb {R} \rightarrow \mathbb {R}$ at the point $x_{0}$ is now found by calculating $f(\langle x_{0},1\rangle )$ using the above arithmetic, which gives $\langle f(x_{0}),f'(x_{0})\rangle$ as the result.

Vector arguments and functions

Multivariate functions can be handled with the same efficiency and mechanisms as univariate functions by adopting a directional derivative operator. That is, if it is sufficient to compute $y'=\nabla f(x)\cdot x'$, the directional derivative $y'\in \mathbb {R} ^{m}$ of $f:\mathbb {R} ^{n}\rightarrow \mathbb {R} ^{m}$ at $x\in \mathbb {R} ^{n}$ in the direction $x'\in \mathbb {R} ^{n}$, this may be calculated as $(\langle y_{1},y'_{1}\rangle ,\ldots ,\langle y_{m},y'_{m}\rangle )=f(\langle x_{1},x'_{1}\rangle ,\ldots ,\langle x_{n},x'_{n}\rangle )$ using the same arithmetic as above. If all the elements of $\nabla f$ are desired, then $n$ function evaluations are required. Note that in many optimization applications, the directional derivative is indeed sufficient.

Higher order differentials

The above arithmetic can be generalized, in the natural way, to calculate parts of the second order and higher derivatives. However, the arithmetic rules quickly grow very complicated: complexity will be quadratic in the highest derivative degree. Instead, truncated Taylor series arithmetic is used. This is possible because the Taylor summands in a Taylor series of a function are products of known coefficients and derivatives of the function. Currently, there exists efficient Hessian automatic differentiation methods that calculate the entire Hessian matrix with a single forward and reverse accumulation. There also exist a number of specialized methods for calculating large sparse Hessian matrices.

Implementation

Forward-mode AD is implemented by a nonstandard interpretation of the program in which real numbers are replaced by dual numbers, constants are lifted to dual numbers with a zero epsilon coefficient, and the numeric primitives are lifted to operate on dual numbers. This nonstandard interpretation is generally implemented using one of two strategies: source code transformation or operator overloading.

Source code transformation (SCT)

Figure 4: Example of how source code transformation could work

The source code for a function is replaced by an automatically generated source code that includes statements for calculating the derivatives interleaved with the original instructions.

Source code transformation can be implemented for all programming languages, and it is also easier for the compiler to do compile time optimizations. However, the implementation of the AD tool itself is more difficult.

Operator overloading (OO)

Figure 5: Example of how operator overloading could work

Operator overloading is a possibility for source code written in a language supporting it. Objects for real numbers and elementary mathematical operations must be overloaded to cater for the augmented arithmetic depicted above. This requires no change in the form or sequence of operations in the original source code for the function to be differentiated, but often requires changes in basic data types for numbers and vectors to support overloading and often also involves the insertion of special flagging operations.

Operator overloading for forward accumulation is easy to implement, and also possible for reverse accumulation. However, current compilers lag behind in optimizing the code when compared to forward accumulation.

Software

• C/C++

Package	License	Approach	Brief Info
ADC Version 4.0	nonfree	OO
Adept	GPL 3	OO	First-order forward and reverse modes. Very fast due to its use of expression templates and an efficient tape structure.
ADIC	free for noncommercial	SCT	forward mode
ADMB	Template:BSD-lic	SCT+OO
ADNumber	dual license	OO	arbitrary order forward/reverse
ADOL-C	CPL 1.0 or GPL 2.0	OO	arbitrary order forward/reverse, part of COIN-OR
AMPL	free for students	SCT
FADBAD++	free for noncommercial	OO	uses operator new
CasADi	LGPL	SCT	Forward/reverse modes, matrix-valued atomic operations.
ceres-solver	BSD	OO	A portable C++ library that allows for modeling and solving large complicated nonlinear least squares problems
CppAD	EPL 1.0 or GPL 3.0	OO	arbitrary order forward/reverse, AD<Base> for arbitrary Base including AD<Other_Base>, part of COIN-OR; can also be used to produce C source code using the CppADCodeGen library.
OpenAD	depends on components	SCT
Sacado	Template:GPL-lic	OO	A part of the Trilinos collection, forward/reverse modes.
Stan	Template:BSD-lic	OO	forward- and reverse-mode automatic differentiation with library of special functions, probability functions, matrix operators, and linear algebra solvers
TAPENADE	Free for noncommercial	SCT
CTaylor	free	OO	truncated taylor series, multi variable, high performance, calculating and storing only potentially nonzero derivatives, calculates higher order derivatives, order of derivatives increases when using matching operations until maximum order (parameter) is reached, example source code and executable available for testing performance

• Fortran

Package	License	Approach	Brief Info
ADF Version 4.0	nonfree	OO
ADIFOR	>>> (free for non-commercial)	SCT
AUTO_DERIV	free for non-commercial	OO
OpenAD	depends on components	SCT
TAPENADE	Free for noncommercial	SCT

• Matlab

Package	License	Approach	Brief Info
AD for MATLAB	Template:GPL-lic	OO	Forward (1st & 2nd derivative, Uses MEX files & Windows DLLs)
Adiff	Template:BSD-lic	OO	Forward (1st derivative)
MAD	Proprietary	OO
ADiMat	?	SCT	Forward (1st & 2nd derivative) & Reverse (1st)

• Python

Package	License	Approach	Brief Info
ad	Template:BSD-lic	OO	first and second-order, reverse accumulation, transparent on-the-fly calculations, basic NumPy support, written in pure python
FuncDesigner	Template:BSD-lic	OO	uses NumPy arrays and SciPy sparse matrices, also allows to solve linear/non-linear/ODE systems and to perform numerical optimizations by OpenOpt
ScientificPython	CeCILL	OO	see modules Scientific.Functions.FirstDerivatives and Scientific.Functions.Derivatives
pycppad	Template:BSD-lic	OO	arbitrary order forward/reverse, implemented as wrapper for CppAD including AD<double> and AD< AD<double> >.
pyadolc	Template:BSD-lic	OO	wrapper for ADOL-C, hence arbitrary order derivatives in the (combined) forward/reverse mode of AD, supports sparsity pattern propagation and sparse derivative computations
uncertainties	Template:BSD-lic	OO	first-order derivatives, reverse mode, transparent calculations
algopy	Template:BSD-lic	OO	same approach as pyadolc and thus compatible, support to differentiate through numerical linear algebra functions like the matrix-matrix product, solution of linear systems, QR and Cholesky decomposition, etc.
pyderiv	Template:GPL-lic	OO	automatic differentiation and (co)variance calculation
CasADi	LGPL	SCT	Python front-end to CasADi. Forward/reverse modes, matrix-valued atomic operations.
Theano	Template:BSD-lic	OO	Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays both on CPU and GPU efficiently.

• .NET

Package	License	Approach	Brief Info
AutoDiff	Template:GPL-lic	OO	Automatic differentiation with C# operators overloading.
FuncLib	MIT	OO	Automatic differentiation and numerical optimization, operator overloading, unlimited order of differentiation, compilation to IL code for very fast evaluation.

• Haskell

Package	License	Approach	Brief Info
ad	Template:BSD-lic	OO	Forward Mode (1st derivative or arbitrary order derivatives via lazy lists and sparse tries) Reverse Mode Combined forward-on-reverse Hessians. Uses Quantification to allow the implementation automatically choose appropriate modes. Quantification prevents perturbation/sensitivity confusion at compile time.
fad	Template:BSD-lic	OO	Forward Mode (lazy list). Quantification prevents perturbation confusion at compile time.
rad	Template:BSD-lic	OO	Reverse Mode. (Subsumed by 'ad'). Quantification prevents sensitivity confusion at compile time.

• Octave

Package	License	Approach	Brief Info
CasADi	LGPL	SCT	Octave front-end to CasADi. Forward/reverse modes, matrix-valued atomic operations.

• Java

Package	License	Approach	Brief Info
JAutoDiff	-	OO	Provides a framework to compute derivatives of functions on arbitrary types of field using generics. Coded in 100% pure Java.
Apache Commons Math	Apache License v2	OO	This class is an implementation of the extension to Rall's numbers described in Dan Kalman's paper
Deriva	Eclipse Public License v1.0	DSL+Code Generation	Deriva automates algorithmic differentiation in Java and Clojure projects. It defines DSL for building extended arithmetic expressions (the extension being support for conditionals, allowing to express non analytic functions). The DSL is used to generate flat byte-code at runtime, providing implementation without overhead of function calls.
Jap	Public	OO/SCT	Jap is a tools using Virtual Operator Overloading for java class. Jap was developed in the thesis of Phuong PHAM-QUANG 2008-2011.

• Clojure

Package	License	Approach	Brief Info
Deriva	Eclipse Public License v1.0	DSL+Code Generation	Deriva automates algorithmic differentiation in Java and Clojure projects. It defines DSL for building extended arithmetic expressions (the extension being support for conditionals, allowing to express non analytic functions). The DSL is used to generate flat byte-code at runtime, providing implementation without overhead of function calls.

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

Literature

• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534
• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534
• One of the biggest reasons investing in a Singapore new launch is an effective things is as a result of it is doable to be lent massive quantities of money at very low interest rates that you should utilize to purchase it. Then, if property values continue to go up, then you'll get a really high return on funding (ROI). Simply make sure you purchase one of the higher properties, reminiscent of the ones at Fernvale the Riverbank or any Singapore landed property Get Earnings by means of Renting
  
  In its statement, the singapore property listing - website link, government claimed that the majority citizens buying their first residence won't be hurt by the new measures. Some concessions can even be prolonged to chose teams of consumers, similar to married couples with a minimum of one Singaporean partner who are purchasing their second property so long as they intend to promote their first residential property. Lower the LTV limit on housing loans granted by monetary establishments regulated by MAS from 70% to 60% for property purchasers who are individuals with a number of outstanding housing loans on the time of the brand new housing purchase. Singapore Property Measures - 30 August 2010 The most popular seek for the number of bedrooms in Singapore is 4, followed by 2 and three. Lush Acres EC @ Sengkang
  
  Discover out more about real estate funding in the area, together with info on international funding incentives and property possession. Many Singaporeans have been investing in property across the causeway in recent years, attracted by comparatively low prices. However, those who need to exit their investments quickly are likely to face significant challenges when trying to sell their property – and could finally be stuck with a property they can't sell. Career improvement programmes, in-house valuation, auctions and administrative help, venture advertising and marketing, skilled talks and traisning are continuously planned for the sales associates to help them obtain better outcomes for his or her shoppers while at Knight Frank Singapore. No change Present Rules
  
  Extending the tax exemption would help. The exemption, which may be as a lot as $2 million per family, covers individuals who negotiate a principal reduction on their existing mortgage, sell their house short (i.e., for lower than the excellent loans), or take part in a foreclosure course of. An extension of theexemption would seem like a common-sense means to assist stabilize the housing market, but the political turmoil around the fiscal-cliff negotiations means widespread sense could not win out. Home Minority Chief Nancy Pelosi (D-Calif.) believes that the mortgage relief provision will be on the table during the grand-cut price talks, in response to communications director Nadeam Elshami. Buying or promoting of blue mild bulbs is unlawful.
  
  A vendor's stamp duty has been launched on industrial property for the primary time, at rates ranging from 5 per cent to 15 per cent. The Authorities might be trying to reassure the market that they aren't in opposition to foreigners and PRs investing in Singapore's property market. They imposed these measures because of extenuating components available in the market." The sale of new dual-key EC models will even be restricted to multi-generational households only. The models have two separate entrances, permitting grandparents, for example, to dwell separately. The vendor's stamp obligation takes effect right this moment and applies to industrial property and plots which might be offered inside three years of the date of buy. JLL named Best Performing Property Brand for second year running
  
  The data offered is for normal info purposes only and isn't supposed to be personalised investment or monetary advice. Motley Fool Singapore contributor Stanley Lim would not personal shares in any corporations talked about. Singapore private home costs increased by 1.eight% within the fourth quarter of 2012, up from 0.6% within the earlier quarter. Resale prices of government-built HDB residences which are usually bought by Singaporeans, elevated by 2.5%, quarter on quarter, the quickest acquire in five quarters. And industrial property, prices are actually double the levels of three years ago. No withholding tax in the event you sell your property. All your local information regarding vital HDB policies, condominium launches, land growth, commercial property and more
  
  There are various methods to go about discovering the precise property. Some local newspapers (together with the Straits Instances ) have categorised property sections and many local property brokers have websites. Now there are some specifics to consider when buying a 'new launch' rental. Intended use of the unit Every sale begins with 10 p.c low cost for finish of season sale; changes to 20 % discount storewide; follows by additional reduction of fiftyand ends with last discount of 70 % or extra. Typically there is even a warehouse sale or transferring out sale with huge mark-down of costs for stock clearance. Deborah Regulation from Expat Realtor shares her property market update, plus prime rental residences and houses at the moment available to lease Esparina EC @ Sengkang

External links

• www.autodiff.org, An "entry site to everything you want to know about automatic differentiation"
• Automatic Differentiation of Parallel OpenMP Programs
• Automatic Differentiation, C++ Templates and Photogrammetry
• Automatic Differentiation, Operator Overloading Approach
• Compute analytic derivatives of any Fortran77, Fortran95, or C program through a web-based interface Automatic Differentiation of Fortran programs
• Description and example code for forward Automatic Differentiation in Scala
• Adjoint Algorithmic Differentiation: Calibration and Implicit Function Theorem