|
|
| Line 1: |
Line 1: |
| In [[mathematics]], '''rejection sampling''' is a basic technique used to generate observations from a [[probability distribution|distribution]]. It is also commonly called the '''acceptance-rejection method''' or "accept-reject algorithm".
| | Anybody who wrote the guide is called Eusebio. South Carolina is his birth place. The favorite hobby for him and his kids is to assist you fish and he's been for a while doing it for quite some time. Filing has been his profession as word spread. Go to his website to discover a out more: http://circuspartypanama.com<br><br>Feel free to visit my blog - [http://circuspartypanama.com clash of clans hack cydia] |
| | |
| Rejection sampling is based on the observation that to sample a [[random variable]] one can sample uniformly from the region under the graph of its density function.<ref name="radford03">{{cite journal
| |
| |first=Radford M. |last=Neal
| |
| |title=Slice Sampling
| |
| |journal=[[Annals of Statistics]]
| |
| |volume=31 |issue=3 |pages=705–767 |year=2003
| |
| |doi=10.1214/aos/1056562461
| |
| |mr=1994729 | zbl = 1051.65007
| |
| }}</ref><ref name="bishop06">{{cite book
| |
| | last = Bishop
| |
| | first = Christopher
| |
| | title = Pattern Recognition and Machine Learning
| |
| | publisher = [[Springer Science+Business Media|Springer]]
| |
| | year = 2006
| |
| | chapter = 11.4: Slice sampling
| |
| | isbn = 0-387-31073-8
| |
| }}</ref>
| |
| | |
| ==Description==
| |
| To visualize the motivation behind rejection sampling, imagine graphing the distribution of a random variable onto a large rectangular board and throwing darts at it. Assume that the darts are uniformly distributed around the board. Now take off (reject) all of the darts that are outside the area under the curve. The remaining darts will be distributed uniformly within the area under the curve, and the x-positions of these darts will be distributed according to the random variable's density. This is because there is the most room for the darts to land where the curve is highest and thus the probability density is greatest.
| |
| | |
| The visualization as just described is equivalent to a particular form of rejection sampling where the proposal distribution is uniform (hence its graph is a rectangle). The general form of rejection sampling assumes that the board is not necessarily rectangular but is shaped according to some distribution that we know how to sample from (e.g. using [[inversion sampling]]), and which is at least as high at every point as the distribution we want to sample from, so that the former completely encloses the latter. (Otherwise, there will be parts of the curved area we want to sample from that can never be reached.) Rejection
| |
| sampling works as follows:
| |
| #Sample a point (an x-position) from the proposal distribution.
| |
| #Draw a vertical line at this x-position, up to the curve of the proposal distribution.
| |
| #Sample uniformly along this line (i.e. uniformly from 0 to the value of the proposal distribution (maximum of the probability density function)). If the sampled value is greater than the value of the desired distribution at this vertical line, return to step 1.
| |
| | |
| Note also that this algorithm can be used to sample from the area under ''any'' curve, regardless of whether the function integrates to 1. In fact, scaling a function by a constant has no effect on the sampled x-positions. This means that the algorithm can be used to sample from a distribution whose [[probability density function]] is only known up to a constant (i.e. whose [[normalizing constant]] is unknown), which is common in [[computational statistics]].
| |
| | |
| ==Examples==
| |
| [[File:Circle sampling.png|right|140px]]
| |
| As a simple geometric example, suppose it is desired to generate a random point within the unit circle. Generate a candidate point <math>(x,y)</math> where <math>x</math> and <math>y</math> are independent uniformly distributed between −1 and 1. If it so happens that <math>x^2+y^2 \leq 1</math> then the point is within the unit circle and should be accepted. If not then this point should be rejected and another candidate should be generated.
| |
| | |
| The [[ziggurat algorithm]], a more advanced example, is used to efficiently generate [[Normal distribution|normally-distributed]] [[pseudorandom number]]s.
| |
| | |
| ==Theory==
| |
| The rejection sampling method generates sampling values from an arbitrary [[probability distribution]] function <math>f(x)</math> by using an instrumental distribution <math>g(x)</math>, under the only restriction that <math>f(x)< M g(x)</math> where <math>M>1</math> is an appropriate bound on <math>f(x)/g(x)</math>.
| |
| | |
| Rejection sampling is usually used in cases where the form of <math>f(x)</math> makes sampling difficult. Instead of sampling directly from the distribution <math>f(x)</math>, we use an envelope distribution <math>Mg(x)</math> where sampling is easier. These samples from <math>Mg(x)</math> are probabilistically accepted or rejected.
| |
| | |
| This method relates to the general field of [[Monte Carlo method|Monte Carlo]] techniques, including [[Markov chain Monte Carlo]] algorithms that also use a proxy distribution to achieve simulation from the target distribution <math>f(x)</math>. It forms the basis for algorithms such as the [[Metropolis algorithm]].
| |
| | |
| The '''unconditional acceptance probability''' is the proportion of proposed samples which are accepted, which is <math>Pr(u<f(x)/Mg(x))</math> <math>= E[f(x)/Mg(x)] = </math> <math> \int(f(x)/Mg(x))g(x)dx = (1/M)</math><math>\int f(x)dx = 1/M</math>. If <math>M</math> is low, fewer samples are rejected, and the required number of samples for the target distribution is obtained more quickly. Because <math>M</math> must be no less than the maximum of <math>f(x)/g(x)</math>, the unconditional acceptance probability is higher the less that ratio varies, however to obtain acceptance probability 1, <math>f(x)=g(x)</math>, which defeats the purpose of sampling.
| |
| | |
| ==Algorithm==
| |
| The algorithm (used by [[John von Neumann]] and dating back to Buffon and [[Buffon's needle|his needle]]) is as follows:
| |
| * Sample <math>x</math> from <math>g(x)</math> and <math>u</math> from <math>U(0,1)</math> (the uniform distribution over the unit interval)
| |
| * Check whether or not <math>u<f(x)/Mg(x)</math>.
| |
| ** If this holds, accept <math>x</math> as a realization of <math>f(x)</math>;
| |
| ** if not, reject the value of <math>x</math> and repeat the sampling step.
| |
| | |
| The validation of this method is the '''envelope principle''': when simulating the pair <math>(x,v=u*Mg(x))</math>, one produces a uniform simulation over the subgraph of <math>Mg(x)</math>. Accepting only pairs such that <math>u<f(x)/Mg(x)</math> then produces pairs <math>(x,v)</math> uniformly distributed over the subgraph of <math>f(x)</math> and thus, marginally, a simulation from <math>f(x)</math>.
| |
| | |
| This means that, with enough replicates, the algorithm generates a sample from the desired distribution <math>f(x)</math>. There are a number of extensions to this algorithm, such as the [[Metropolis algorithm]].
| |
| | |
| ==Drawbacks==
| |
| | |
| Rejection sampling can lead to a lot of unwanted samples being taken if the function being sampled is highly concentrated in a certain region, for example a function that has a spike at some location. For many distributions, this problem can be solved using an adaptive extension (see [[#Adaptive rejection sampling|adaptive rejection sampling]]). In addition, as the dimensions of the problem get larger, the ratio of the embedded volume to the "corners" of the embedding volume tends towards zero, thus a lot of rejections can take place before a useful sample is generated, thus making the algorithm inefficient and impractical. See [[curse of dimensionality]]. In high dimensions, it is necessary to use a different approach, typically a [[Markov chain Monte Carlo]] method such as [[Metropolis sampling]] or [[Gibbs sampling]]. (However, Gibbs sampling, which breaks down a multi-dimensional sampling problem into a series of low-dimensional samples, may use rejection sampling as one of its steps.)
| |
| | |
| ==Adaptive rejection sampling==
| |
| | |
| For many distributions, finding a proposal distribution that includes the given distribution without a lot of wasted space is difficult. An extension of rejection sampling that can be used to overcome this difficulty and efficiently sample from a large number of distributions (provided that they are [[Logarithmically concave function|log-concave]], which is in fact the case for most of the common distributions) is known as '''adaptive rejection sampling'''. The basic idea is to model the proposal distribution using a set of piecewise [[exponential distribution]]s (i.e. segments of one or more exponential distributions, attached end to end). This can be easier visualized in log space (i.e. by looking at the logarithm of the distribution). The logarithm of an exponential distribution is a straight line, and hence this method essentially involves enclosing the logarithm of the density in a series of line segments. This is the source of the log-concave restriction: if a distribution is log-concave, then its logarithm is concave (shaped like an upside-down U), meaning that a line segment tangent to the curve will always pass over the curve. The method essentially involves successively determining an envelope of straight-line segments that approximates the logarithm better and better while still remaining above the curve, starting with a fixed number of segments (possibly just a single tangent line). Any time we choose a point that is rejected, we tighten the envelope with another line segment that is tangent to the curve at the point with the same x-coordinate as the chosen point.
| |
| | |
| ==See also==
| |
| *[[Inverse transform sampling]]
| |
| *[[Pseudo-random number sampling]]
| |
| | |
| ==References==
| |
| | |
| {{reflist}}
| |
| *Robert, C.P. and Casella, G. "Monte Carlo Statistical Methods" (second edition). New York: Springer-Verlag, 2004.
| |
| *J. von Neumann, "Various techniques used in connection with random digits. Monte Carlo methods", Nat. Bureau Standards, 12 (1951), pp. 36–38.
| |
| | |
| [[Category:Monte Carlo methods]]
| |
| [[Category:Non-uniform random numbers]]
| |
Anybody who wrote the guide is called Eusebio. South Carolina is his birth place. The favorite hobby for him and his kids is to assist you fish and he's been for a while doing it for quite some time. Filing has been his profession as word spread. Go to his website to discover a out more: http://circuspartypanama.com
Feel free to visit my blog - clash of clans hack cydia