Kinetic width: Difference between revisions

From formulasearchengine
Jump to navigation Jump to search
en>Dthomsen8
Reviewed by Dthomsen8
 
en>Mild Bill Hiccup
m →‎2D Case: Cleaned up using AutoEd; an → a
 
Line 1: Line 1:
{{context|date=May 2012}}
Four or five years ago, a reader of some of my columns bought the domain name jamesaltucher.com and gave it to me as a birthday gift. It was a total surprise to me. I didn't even know the reader. I hope one day we meet.<br>Two years ago a friend of mine, Tim Sykes, insisted I had to have a blog. He set it up for me. He even wrote the "About Me". I didn't want a blog. I had nothing to say. But about 6 or 7 months ago I decided I wanted to take this blog seriously. I kept putting off changing the "About Me" which was no longer really about me and maybe never was.<br>A few weeks ago I did a chapter in one of the books in Seth Godin's "The Domino Project". The book is out and called "No Idling". Mohit Pewar organized it (here's Mohit's blog) and sent me a bunch of questions recently. It's intended to be an interview on his blog but I hope Mohit forgives me because I want to use it as my new "About Me" also.<br>1. You are a trader, investor, writer, and entrepreneur? Which of these roles you enjoy the most and why?<br>When I first moved to New York City in 1994 I wanted to be everything to everyone. I had spent the six years prior to that writing a bunch of unpublished novels and unpublished short stories. I must've sent out 100s of stories to literary journals. I got form rejections from every publisher, journal, and agent I sent my novels and stories to.<br>Now, in 1994, everything was possible. The money was in NYC. Media was here. I lived in my 10�10 room and pulled suits out of a garbage bag every morning but it didn't matter...the internet was revving up and I knew how to build a website. One of the few in the city. My sister warned me though: nobody here is your friend. Everybody wants something<br>
In [[statistics]] and [[machine learning]], a '''Bayesian interpretation of regularization''' for [[kernel methods]] is often useful. Kernel methods are central to both the [[Regularization (mathematics)|regularization]] and the [[Bayesian probability|Bayesian]] point of views in machine learning. In regularization they are a natural choice for the [[Statistical learning theory#Formal Description|hypothesis space]] and the regularization functional through the notion of [[reproducing kernel Hilbert space]]s. In Bayesian probability they are a key component of [[Gaussian processes]], where the kernel function is known as the covariance function.  Kernel methods have traditionally been used in [[supervised learning]] problems where the ''input space'' is usually a ''space of vectors'' while the ''output space'' is a ''space of scalars''. More recently these methods have been extended to problems that deal with multiple outputs such as in [[multi-task learning]].<ref name=AlvRosLaw11>{{cite journal|last=Álvarez|first=Mauricio A.|coauthors=Rosasco, Lorenzo; Lawrence, Neil D.|title=Kernels for Vector-Valued Functions: A Review|journal=ArXiv e-prints|date=June 2011}}</ref>
And I wanted something. I wanted the fleeting feelings of success, for the first time ever, in order to feel better about myself. I wanted a girl next to me. I wanted to build and sell companies and finally prove to everyone I was the smartest. I wanted to do a TV show. I wanted to write books<br>
But everything involved having a master. Clients. Employers. Investors. Publishers. The market (the deadliest master of all). Employees. I was a slave to everyone for so many years. And the more shackles I had on, the lonelier I got<br>
Much of the time, even when I had those moments of success, I didn't know how to turn it into a better life. I felt ugly and then later, I felt stupid when I would let the success dribble away down the sink<br>
I love writing because every now and then that ugliness turns into honesty. When I write, I'm only a slave to myself. When I do all of those other things you ask about, I'm a slave to everyone else<br>
Some links<br>
33 Unusual Tips to Being a Better Write<br>
"The Tooth<br>
(one of my favorite posts on my blog<br>
2. What inspires you to get up and start working/writing every day<br>
The other day I had breakfast with a fascinating guy who had just sold a piece of his fund of funds. He told me what "fracking" was and how the US was going to be a major oil player again. We spoke for two hours about a wide range of topics, including what happens when we can finally implant a google chip in our brains<br>
After that I had to go onto NPR because I firmly believe that in one important respect we are degenerating as a country - we are graduating a generation of indentured servants who will spend 50 years or more paying down their student debt rather than starting companies and curing cancer. So maybe I made a difference<br>
Then I had lunch with a guy I hadn't seen in ten years. In those ten years he had gone to jail and now I was finally taking the time to forgive him for something he never did to me. I felt bad I hadn't helped him when he was at his low point. Then I came home and watched my kid play clarinet at her school. Then I read until I fell asleep. Today I did nothing but write. Both days inspired me<br>
It also inspires me that I'm being asked these questions. Whenever anyone asks me to do anything I'm infinitely grateful. Why me? I feel lucky. I like it when someone cares what I think. I'll write and do things as long as anyone cares. I honestly probably wouldn't write if nobody cared. I don't have enough humility for that, I'm ashamed to admit<br>
3. Your new book "How to be the luckiest person alive" has just come out. What is it about<br>
When I was a kid I thought I needed certain things: a college education from a great school, a great home, a lot of money, someone who would love me with ease. I wanted people to think I was smart. I wanted people to think I was even special.  And as I grew older more and more goals got added to the list: a high chess rating, a published book, perfect weather, good friends,  respect in various fields, etc. I lied to myself that I needed these things to be happy. The world was going to work hard to give me these things, I thought. But it turned out the world owed me no favors<br>
And gradually, over time, I lost everything I had ever gained. Several times.  I've paced at night so many times wondering what the hell was I going to do next or trying not to care. The book is about regaining your sanity, regaining your happiness, finding luck in all the little pockets of life that people forget about. It's about turning away from the religion you've been hypnotized into believing into the religion you can find inside yourself every moment of the day<br>
[Note: in a few days I'm going to do a post on self-publishing and also how to get the ebook for free. The link above is to the paperback. Kindle should be ready soon also.<br>
Related link: Why I Write Books Even Though I've Lost Money On Every Book I've Ever Writte<br>
4. Is it possible to accelerate success? If yes, how<br>
Yes, and it's the only way I know actually to achieve success. It's by following the Daily Practice I outline in this post:<br>
It's the only way I know to exercise every muscle from the inside of you to the outside of you. I firmly believe that happiness starts with that practice<br>
5. You say that discipline, persistence and psychology are important if one has to achieve success. How can one work on improving "psychology" part<br>
Success doesn't really mean anything. People want to be happy in a harsh and unforgiving world. It's very difficult. We're so lucky most of us live in countries without [http://Dict.leo.org/?search=major+wars major wars]. Our kids aren't getting killed by random gunfire. We all have cell phones. We all can communicate with each other on the Internet. We have Google to catalog every piece of information in history!  We are so amazingly lucky already<br>
How can it be I was so lucky to be born into such a body? In New York City of all places? Just by being born in such a way on this planet was an amazing success<br>
So what else is there? The fact is that most of us, including me, have a hard time being happy with such ready-made success. We quickly adapt and want so much more out of life. It's not wars or disease that kill us. It's the minor inconveniences that add up in life. It's the times we feel slighted or betrayed. Or even slightly betrayed. Or overcharged. Or we miss a train. Or it's raining today. Or the dishwasher doesn't work. Or the supermarket doesn't have the food we like. We forget how good the snow tasted when we were kids. Now we want gourmet food at every meal<br>
Taking a step back, doing the Daily Practice I outline in the question above. For me, the results of that bring me happiness. That's success. Today. And hopefully tomorrow<br>
6. You advocate not sending kids to college. What if kids grow up and then blame their parents about not letting them get a college education<br>
I went to one of my kid's music recitals yesterday. She was happy to see me. I hugged her afterwards. She played "the star wars theme" on the clarinet. I wish I could've played that for my parents. My other daughter has a dance recital in a few weeks. I tried to give her tips but she laughed at me. I was quite the breakdancer in my youth. The nerdiest breakdancer on the planet. I want to be present for them. To love them. To let them always know that in their own dark moments, they know I will listen to them. I love them. Even when they cry and don't always agree with me. Even when they laugh at me because sometimes I act like a clown<br>
Later, if they want to blame me for anything at all then I will still love them. That's my "what if"<br>
Two posts<br>
I want my daughters to be lesbian<br>
Advice I want to give my daughter<br><br>


In this article we analyze the connections between the regularization and Bayesian point of views for kernel methods in the case of scalar outputs.  A mathematical equivalence between the regularization and the Bayesian point of views is easily proved in cases where the reproducing kernel Hilbert space is ''finite-dimensional''. The infinite-dimensional case raises subtle mathematical issues; we will consider here the finite-dimensional case. We start with a brief review of the main ideas underlying kernel methods for scalar learning, and briefly introduce the concepts of regularization and Gaussian processes. We then show how both point of views arrive at essentially equivalent estimators, and show the connection that ties them together.


==The Supervised Learning Problem==
7. Four of your favorite posts from The Altucher Confidential<br>
 
As soon as I publish a post I get scared to death. Is it good? Will people re-tweet? Will one part of the audience of this blog like it at the expense of another part of the audience. Will I get Facebook Likes? I have to stop clinging to these things but you also need to respect the audience. I don't know. It's a little bit confusing to me. I don't have the confidence of a real writer yet<br>
The classical [[supervised learning]] problem requires estimating the output for some new input point <math>\mathbf{x}'</math> by learning a scalar-valued estimator <math>\hat{f}(\mathbf{x}')</math> on the basis of a training set <math>S</math> consisting of <math>n</math> input-output pairs, <math>S = (\mathbf{X},\mathbf{Y}) = (\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_n,y_n)</math>.<ref name=Vap98>{{cite book|last=Vapnik|first=Vladimir|title=Statistical learning theory|year=1998|publisher=Wiley|isbn=9780471030034|url=http://books.google.com/books?id=GowoAQAAMAAJ&q=statistical+learning+theory&dq=statistical+learning+theory&hl=en&sa=X&ei=HruyT66kOoKhgwf3reSXCQ&ved=0CDsQ6AEwAA}}</ref><ref name=HasTibFri09>{{cite book|last=Hastie|first=Trevor|title=The Elements of Statistical Learning: Data Mining, Inference and Prediction|year=2009|publisher=Springer|isbn=9780387848570|edition=2, illustrated|coauthors=Tibshirani, Robert; Friedman, Jerome H.}}</ref><ref name=Bis09>{{cite book|last=Bishop|first=Christopher M.|title=Pattern recognition and machine learning|year=2009|publisher=Springer|isbn=9780387310732}}</ref>  Given a symmetric and positive bivariate function <math>k(\cdot,\cdot)</math> called a ''kernel'', one of the most popular estimators in machine learning is given by
Here are four of my favorites<br>
 
How I screwed Yasser Arafat out of $2mm (and lost another $100mm in the process<br>
{{NumBlk|:|<math>
It's Your Fault<br>
\hat{f}(\mathbf{x}') = \mathbf{k}^\top(\mathbf{K} + \lambda n \mathbf{I})^{-1} \mathbf{Y},
I'm Guilty of Torturing Wome<br>
</math>|{{EquationRef|1}}}}
The Girl Whose Name Was a Curs<br>
 
Although these three are favorites I really don't post anything unless it's my favorite of that moment<br>
where <math>\mathbf{K} \equiv k(\mathbf{X},\mathbf{X})</math> is the kernel matrix with entries <math>\mathbf{K}_{ij} = k(\mathbf{x}_i,\mathbf{x}_j)</math>, <math> \mathbf{k} = [k(\mathbf{x}_1,\mathbf{x}'),\ldots,k(\mathbf{x}_n,\mathbf{x}')]^\top</math>, and <math>\mathbf{Y} = [y_1,\ldots,y_n]^\top</math>.  We will see how this estimator can be derived both from a regularization and a Bayesian perspective.
8. 3 must-read books for aspiring entrepreneurs<br>
 
The key in an entrepreneur book: you want to learn business. You want to learn how to honestly communicate with your customers. You want to stand out<br>
==A Regularization Perspective==
The Essays of Warren Buffett by Lawrence Cunningha<br>
 
"The Thank you Economy" by Gary Vaynerchu<br>
The main assumption in the regularization perspective is that the set of functions <math>\mathcal{F}</math> is assumed to belong to a reproducing kernel Hilbert space <math>\mathcal{H}_k</math>.<ref name=Vap98 /><ref name=Wah90 /><ref name=SchSmo02>{{cite book|last=Schölkopf|first=Bernhard|title=Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond|year=2002|publisher=MIT Press|isbn=9780262194754|coauthors=Smola, Alexander J.}}</ref><ref name=GirPog90>{{cite journal|last=Girosi|first=F.|coauthors=Poggio, T.|title=Networks and the best approximation property|journal=Biological Cybernetics|year=1990|volume=63|issue=3|pages=169–176|publisher=Springer}}</ref>
"Purple cow" by Seth Godi<br>
 
9. I love your writing, so do so many others out there. Who are your favorite writers<br>
===Reproducing Kernel Hilbert Space===
"Jesus's Son" by Denis Johnson is the best collection of short stories ever written. I'm afraid I really don't like his novels though<br>
 
"Tangents" by M. Prado. A beautiful series of graphic stories about relationships<br>
A [[reproducing kernel Hilbert space]] (RKHS) <math>\mathcal{H}_k</math> is a [[Hilbert space]] of functions defined by a [[Symmetry in mathematics|symmetric]], [[positive-definite function]] <math>k : \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}</math> called the ''reproducing kernel'' such that the function <math>k(\mathbf{x},\cdot)</math> belongs to <math>\mathcal{H}_k</math> for all <math>\mathbf{x} \in \mathcal{X}</math>.<ref name=Aro50>{{cite journal|last=Aronszajn|first=N|title=Theory of Reproducing Kernels|journal=Transactions of the American Mathematical Society|date=May 1950|volume=68|issue=3|pages=337–404}}</ref><ref name=Sch64>{{cite journal|last=Schwartz|first=Laurent|title=Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés (noyaux reproduisants)|journal=Journal d'analyse mathématique|year=1964|volume=13|issue=1|pages=115–256|publisher=Springer}}</ref><ref name=CucSma01>{{cite journal|last=Cucker|first=Felipe|coauthors=Smale, Steve|title=On the mathematical foundations of learning|journal=Bulletin of the American Mathematical Society|date=October 5, 2001|volume=39|issue=1|pages=1–49}}</ref> There are three main properties make an RKHS appealing:
Other writers: Miranda July, Ariel Leve, Mary Gaitskill, Charles Bukowski, [http://www.pcs-systems.co.uk/Images/celinebag.aspx Celine Bags Outlet], Sam Lipsyte, William Vollmann, Raymond Carver. Arthur Nersesian. Stephen Dubner<br>
 
Many writers are only really good storytellers. Most writers come out of a cardboard factory MFA system and lack a real voice. A real voice is where every word exposes ten levels of hypocrisy in the world and brings us all the way back to see reality. The writers above have their own voices, their own pains, and their unique ways of expressing those pains. Some of them are funny. Some a little more dark. I wish I could write 1/10 as good as any of them<br><br>
1. The ''reproducing property'', which gives name to the space,
10. You are a prolific writer. Do you have any hacks that help you write a lot in little time<br>
 
Coffee, plus everything else coffee does for you first thing in the morning<br>
<math>
Only write about things you either love or hate. But if you hate something, try to find a tiny gem buried in the bag of dirt so you can reach in when nobody is looking and put that gem in your pocket. Stealing a diamond in all the shit around us and then giving it away for free via writing is a nice little hack, Being fearless precisely when you are most scared is the best hack<br><br>
f(\mathbf{x}) = \langle f,k(\mathbf{x},\cdot) \rangle_k, \quad \forall \ f \in \mathcal{H}_k,
11. I totally get and love your idea about bleeding as a writer, appreciate if you share more with the readers of this blog<br>
</math>
Most people worry about what other people think of them. Most people worry about their health. Most people are at a crossroads and don't know how to take the next step and which road to take it on. Everyone is in a perpetual state of 'where do I put my foot next'. Nobody, including me, can avoid that<br>
 
You and I both need to wash our faces in the morning, brush our teeth, shower, shit, eat, fight the weather, fight the colds that want to attack us if we're not ready. Fight loneliness or learn how to love and appreciate the people who want to love you back. And learn how to forgive and love the people who are even more stupid and cruel than we are. We're afraid to tell each other these things because they are all both disgusting and true<br>
where <math>\langle \cdot,\cdot \rangle_k</math> is the inner product in <math>\mathcal{H}_k</math>.
You and I both have the same color blood. If I cut my wrist open you can see the color of my blood. You look at it and see that it's the same color as yours. We have something in common. It doesn't have to be shameful. It's just red. Now we're friends. No matter whom you are or where you are from. I didn't have to lie to you to get you to be my friend<br>
 
Related Links<br>
2. Functions in an RKHS are in the closure of the linear combination of the kernel at given points,  
How to be a Psychic in Ten Easy Lesson<br>
 
My New Year's Resolution in 199<br>
<math>
12. What is your advice for young entrepreneurs<br>
f(\mathbf{x}) = \sum_i k(\mathbf{x}_i,\mathbf{x})c_i
Only build something you really want to use yourself. There's got to be one thing you are completely desperate for and no matter where you look you can't find it. Nobody has invented it yet. So there you go - you invent it. If there's other people like you, you have a business. Else. You fail. Then do it again. Until it works. One day it will<br>
</math>.
Follow these 100 Rules<br>
 
The 100 Rules for Being a Good Entrepreneur<br>
This allows the construction in a unified framework of both linear and generalized linear models.
And, in particular this<br>
 
The Easiest Way to Succeed as an Entrepreneu<br>
3. The norm in an RKHS can be written as
In my just released book I have more chapters on my experiences as an entrepreneur<br>
 
13. I advocate the concept of working at a job while building your business. You have of course lived it. Now as you look back, what is your take on this? Is it possible to make it work while sailing on two boats<br><br>
<math>\|f\|_k = \sum_{i,j} k(\mathbf{x}_i,\mathbf{x}_j) c_i c_j
Your boss wants everything out of you. He wants you to work 80 hours a week. He wants to look good taking credit for your work. He wants your infinite loyalty. So you need something back<br>
</math>
Exploit your employer. It's the best way to get good experience, clients, contacts. It's a legal way to steal. It's a fast way to be an entrepreneur because you see what large companies with infinite money are willing to pay for. If you can provide that, you make millions. It's how many great businesses have started and will always start. It's how every exit I've had started<br><br>
 
14. Who is a "person with true moral fiber"? In current times are there any role models who are people with true moral fiber<br>
and is a natural measure of how ''complex'' the function is.
I don't really know the answer. I think I know a few people like that. I hope I'm someone like that. And I pray to god the people I'm invested in are like that and my family is like that<br>
 
I find most people to be largely mean and stupid, a vile combination. It's not that I'm pessimistic or cynical. I'm very much an optimist. It's just reality. Open the newspaper or turn on the TV and watch these people<br>
===The Regularized Functional===
Moral fiber atrophies more quickly than any muscle on the body. An exercise I do every morning is to promise myself that "I'm going to save a life today" and then leave it in the hands of the Universe to direct me how I can best do that. Through that little exercise plus the Daily Practice described above I hope to keep regenerating that fiber<br>
 
15.  Your message to the readers of this blog<br>
The estimator is derived as the minimizer of the regularized functional
Skip dinner. But follow me on Twitter.<br>
 
Read more posts on The Altucher Confidential �
{{NumBlk|:|<math>
More from The Altucher Confidentia<br>
\frac{1}{n} \sum_{i=1}^{n}(f(\mathbf{x}_i)-y_i)^2 + \lambda \|f\|_k^2,
Life is Like a Game. Here�s How You Master ANY Gam<br><br>
</math>|{{EquationRef|2}}}}
Step By Step Guide to Make $10 Million And Then Totally Blow <br><br>
 
Can You Do One Page a Day?
where <math>f \in \mathcal{H}_k</math> and <math>\|\cdot\|_k</math> is the norm in <math>\mathcal{H}_k</math>. The first term in this functional, which measures the average of the squares of the errors between the <math>f(\mathbf{x}_i)</math> and the <math>y_i</math>, is called the ''empirical risk'' and represents the cost we pay by predicting <math>f(\mathbf{x}_i)</math> for the true value <math>y_i</math>. The second term in the functional is the squared norm in a RKHS multiplied by a weight <math>\lambda</math> and serves the purpose of stabilizing the problem<ref name=Wah90 /><ref name=GirPog90 /> as well as of adding a trade-off between fitting and complexity of the estimator.<ref name=Vap98 /> The weight <math>\lambda</math>, called the ''regularizer'', determines the degree to which instability and complexity of the estimator should be penalized (higher penalty for increasing value of <math>\lambda</math>).
 
===Derivation of the Estimator===
 
The explicit form of the estimator in equation ({{EquationNote|1}}) is derived in two steps.  First, the representer theorem<ref name=KimWha70>{{cite journal|last=Kimeldorf|first=George S.|coauthors=Wahba, Grace|title=A correspondence between Bayesian estimation on stochastic processes and smoothing by splines|journal=The Annals of Mathematical Statistics|year=1970|volume=41|issue=2|pages=495–502|doi=10.1214/aoms/1177697089}}</ref><ref name=SchHerSmo01>{{cite journal|last=Schölkopf|first=Bernhard|coauthors=Herbrich, Ralf; Smola, Alex J.|title=A Generalized Representer Theorem|journal=COLT/EuroCOLT 2001, LNCS|year=2001|volume=2111/2001|pages=416–426|doi=10.1007/3-540-44581-1_27}}</ref><ref name=DevEtal04>{{cite journal|last=De Vito|first=Ernesto|coauthors=Rosasco, Lorenzo; Caponnetto, Andrea; Piana, Michele; Verri, Alessandro|title=Some Properties of Regularized Kernel Methods|journal=Journal of Machine Learning Research|date=October 2004|volume=5|pages=1363–1390}}</ref> states that the minimizer of the functional ({{EquationNote|2}}) can always be written as a linear combination of the kernels centered at the training-set points,
 
{{NumBlk|:|<math>
\hat{f}(\mathbf{x}') = \sum_{i=1}^n c_i k(\mathbf{x}_i,\mathbf{x}') = \mathbf{k}^\top \mathbf{c},
</math>|{{EquationRef|3}}}}
 
for some <math>\mathbf{c} \in \mathbb{R}^n</math>. The explicit form of the coefficients <math>\mathbf{c} = [c_1,\ldots,c_n]^\top</math> can be found by substituting for <math>f(\cdot)</math> in the functional ({{EquationNote|2}}).  For a function of the form in equation ({{EquationNote|3}}), we have that
 
<math>\begin{align}
\|f\|_k^2 & = \langle f,f \rangle_k, \\
& = \left\langle \sum_{i=1}^N c_i k(\mathbf{x}_i,\cdot), \sum_{j=1}^N c_j k(\mathbf{x}_j,\cdot) \right\rangle_k, \\
& = \sum_{i=1}^N \sum_{j=1}^N c_i c_j \langle k(\mathbf{x}_i,\cdot), k(\mathbf{x}_j,\cdot) \rangle_k, \\
& = \sum_{i=1}^N \sum_{j=1}^N c_i c_j k(\mathbf{x}_i,\mathbf{x}_j), \\
& = \mathbf{c}^\top \mathbf{K} \mathbf{c}.
\end{align}</math>
 
We can rewrite the functional ({{EquationNote|2}}) as
 
<math>
\frac{1}{n} \| \mathbf{y} - \mathbf{K} \mathbf{c} \|^2 + \lambda \mathbf{c}^\top \mathbf{K} \mathbf{c}.
</math>
 
This functional is convex in <math>\mathbf{c}</math> and therefore we can find its minimum by setting the gradient with respect to <math>\mathbf{c}</math> to zero,
 
<math>\begin{align}
-\frac{1}{n} \mathbf{K} (\mathbf{Y} - \mathbf{K} \mathbf{c}) + \lambda \mathbf{K} \mathbf{c} & = 0, \\
(\mathbf{K} + \lambda n \mathbf{I}) \mathbf{c} & = \mathbf{Y}, \\
\mathbf{c} & = (\mathbf{K} + \lambda n \mathbf{I})^{-1} \mathbf{Y}.
\end{align}</math>
 
Substituting this expression for the coefficients in equation ({{EquationNote|3}}), we obtain the estimator stated previously in equation ({{EquationNote|1}}),
 
<math>
\hat{f}(\mathbf{x}') = \mathbf{k}^\top(\mathbf{K} + \lambda n \mathbf{I})^{-1} \mathbf{Y}.
</math>
 
==A Bayesian Perspective==
 
The notion of a kernel plays a crucial role in Bayesian probability as the covariance function of a stochastic process called the ''[[Gaussian process]]''.
 
===A Review of Bayesian Probability===
 
As part of the Bayesian framework, the Gaussian process specifies the [[Prior probability|''prior distribution'']] that describes the prior beliefs about the properties of the function being modeled. These beliefs are updated after taking into account observational data by means of a [[Likelihood function|''likelihood function'']] that relates the prior beliefs to the observations.  Taken together, the prior and likelihood lead to an updated distribution called the [[Posterior probability|''posterior distribution'']] that is customarily used for predicting test cases.
 
===The Gaussian Process===
 
A [[Gaussian process]] (GP) is a stochastic process in which any finite number of random variables that are sampled follow a joint [[Multivariate normal distribution|Normal distribution]].<ref name=RasWil06 />  The mean vector and covariance matrix of the Gaussian distribution completely specify the GP.  GPs are usually used as a priori distribution for functions, and as such the mean vector and covariance matrix can be viewed as functions, where the covariance function is also called the ''kernel'' of the GP. Let a function <math>f</math> follow a Gaussian process with mean function <math>m</math> and kernel function <math>k</math>,
 
<math>
f \sim \mathcal{GP}(m,k).
</math>
 
In terms of the underlying Gaussian distribution, we have that for any finite set <math>\mathbf{X} = \{\mathbf{x}_i\}_{i=1}^{n}</math> if we let <math>f(\mathbf{X}) = [f(\mathbf{x}_1),\ldots,f(\mathbf{x}_n)]^\top</math> then
 
<math>
f(\mathbf{X}) \sim \mathcal{N}(\mathbf{m},\mathbf{K}),
</math>
 
where <math>\mathbf{m} = m(\mathbf{X}) = [m(\mathbf{x}_1),\ldots,m(\mathbf{x}_N)]^\top</math> is the mean vector and <math>\mathbf{K} = k(\mathbf{X},\mathbf{X})</math> is the covariance matrix of the multivariate Gaussian distribution.
 
===Derivation of the Estimator===
 
In a regression context, the likelihood function is usually assumed to be a Gaussian distribution and the observations to be independent and identically distributed (iid),
 
<math>
p(y|f,\mathbf{x},\sigma^2) = \mathcal{N}(f(\mathbf{x}),\sigma^2).
</math>
 
This assumption corresponds to the observations being corrupted with zero-mean Gaussian noise with variance <math>\sigma^2</math>. The iid assumption makes it possible to factorize the likelihood function over the data points given the set of inputs <math>\mathbf{X}</math> and the variance of the noise <math>\sigma^2</math>, and thus the posterior distribution can be computed analytically. For a test input vector <math>\mathbf{x}'</math>, given the training data <math>S = \{\mathbf{X},\mathbf{Y}\}</math>, the posterior distribution is given by
 
<math>
p(f(\mathbf{x}')|S,\mathbf{x}',\boldsymbol{\phi}) = \mathcal{N}(m(\mathbf{x}'),\sigma^2(\mathbf{x}')),
</math>
 
where <math>\boldsymbol{\phi}</math> denotes the set of parameters which include the variance of the noise <math>\sigma^2</math> and any parameters from the covariance function <math>k</math> and where
 
<math>\begin{align}
m(\mathbf{x}') & = \mathbf{k}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{Y}, \\
\sigma^2(\mathbf{x}') & = k(\mathbf{x}',\mathbf{x}') - \mathbf{k}^\top (\mathbf{K} + \sigma^2 \mathbf{I})^{-1} \mathbf{k}.
\end{align}</math>
 
==The Connection Between Regularization and Bayes==
 
A connection between regularization theory and Bayesian theory can only be achieved in the case of ''finite dimensional RKHS''. Under this assumption, regularization theory and Bayesian theory are connected through Gaussian process prediction.<ref name=Wah90>{{cite book|last=Wahba|first=Grace|title=Spline models for observational data|year=1990|publisher=SIAM}}</ref><ref name=RasWil06>{{cite book|last=Rasmussen|first=Carl Edward|title=Gaussian Processes for Machine Learning|year=2006|publisher=The MIT Press|isbn=0-262-18253-X|url=http://www.gaussianprocess.org/gpml/|coauthors=Williams, Christopher K. I.}}</ref>  
 
In the finite dimensional case, every RKHS can be described in terms of a feature map <math>\Phi : \mathcal{X} \rightarrow \mathbb{R}^p</math> such that<ref name=Vap98 />  
 
<math>
k(\mathbf{x},\mathbf{x}') = \sum_{i=1}^p \Phi^i(\mathbf{x})\Phi^i(\mathbf{x}').
</math>
 
Functions in the RKHS with kernel <math>\mathbf{K}</math> can be then be written as
 
<math>
f_{\mathbf{w}}(\mathbf{x}) = \sum_{i=1}^p \mathbf{w}^i \Phi^i(\mathbf{x}) = \langle \mathbf{w},\Phi(\mathbf{x}) \rangle,
</math>
 
and we also have that
 
<math>
\|f_{\mathbf{w}} \|_k = \|\mathbf{w}\|.
</math>
 
We can now build a Gaussian process by assuming <math> \mathbf{w} = [w^1,\ldots,w^p]^\top </math> to be distributed according to a multivariate Gaussian distribution with zero mean and identity covariance matrix,
 
<math>
\mathbf{w} \sim \mathcal{N}(0,\mathbf{I}) \propto \exp(-\|\mathbf{w}\|^2).
</math>
 
If we assume a Gaussian likelihood we have
 
<math>
P(\mathbf{Y}|\mathbf{X},f) = \mathcal{N}(f(\mathbf{X}),\sigma^2 \mathbf{I}) \propto \exp\left(-\frac{1}{\sigma^2} \| f_{\mathbf{w}}(\mathbf{X}) - \mathbf{Y} \|^2\right),
</math>
 
where <math> f_{\mathbf{w}}(\mathbf{X}) = (\langle\mathbf{w},\Phi(\mathbf{x}_1)\rangle,\ldots,\langle\mathbf{w},\Phi(\mathbf{x}_n \rangle) </math>. The resulting posterior distribution is the given by
 
<math>
P(f|\mathbf{X},\mathbf{Y}) \propto \exp\left(-\frac{1}{\sigma^2} \|f_{\mathbf{w}}(\mathbf{X}) - \mathbf{Y}\|_n^2 + \|\mathbf{w}\|^2\right)
</math>
 
We can see that a ''maximum posterior (MAP)'' estimate is equivalent to the minimization problem defining [[Tikhonov regularization]], where in the Bayesian case the regularization parameter is related to the noise variance.
 
From a philosophical perspective, the loss function in a regularization setting plays a different role than the likelihood function in the Bayesian setting. Whereas the loss function measures the error that is incurred when predicting <math>f(\mathbf{x})</math> in place of <math>y</math>, the likelihood function measures how likely the observations are from the model that was assumed to be true in the generative process. From a mathematical perspective, however, the formulations of the regularization and Bayesian frameworks make the loss function and the likelihood function to have the same mathematical role of promoting the inference of functions <math>f</math> that approximate the labels <math>y</math> as much as possible.
 
==References==
{{Reflist}}
 
[[Category:Mathematical analysis]]
[[Category:Machine learning algorithms]]

Latest revision as of 03:58, 14 July 2014

Four or five years ago, a reader of some of my columns bought the domain name jamesaltucher.com and gave it to me as a birthday gift. It was a total surprise to me. I didn't even know the reader. I hope one day we meet.
Two years ago a friend of mine, Tim Sykes, insisted I had to have a blog. He set it up for me. He even wrote the "About Me". I didn't want a blog. I had nothing to say. But about 6 or 7 months ago I decided I wanted to take this blog seriously. I kept putting off changing the "About Me" which was no longer really about me and maybe never was.
A few weeks ago I did a chapter in one of the books in Seth Godin's "The Domino Project". The book is out and called "No Idling". Mohit Pewar organized it (here's Mohit's blog) and sent me a bunch of questions recently. It's intended to be an interview on his blog but I hope Mohit forgives me because I want to use it as my new "About Me" also.
1. You are a trader, investor, writer, and entrepreneur? Which of these roles you enjoy the most and why?
When I first moved to New York City in 1994 I wanted to be everything to everyone. I had spent the six years prior to that writing a bunch of unpublished novels and unpublished short stories. I must've sent out 100s of stories to literary journals. I got form rejections from every publisher, journal, and agent I sent my novels and stories to.
Now, in 1994, everything was possible. The money was in NYC. Media was here. I lived in my 10�10 room and pulled suits out of a garbage bag every morning but it didn't matter...the internet was revving up and I knew how to build a website. One of the few in the city. My sister warned me though: nobody here is your friend. Everybody wants something
And I wanted something. I wanted the fleeting feelings of success, for the first time ever, in order to feel better about myself. I wanted a girl next to me. I wanted to build and sell companies and finally prove to everyone I was the smartest. I wanted to do a TV show. I wanted to write books
But everything involved having a master. Clients. Employers. Investors. Publishers. The market (the deadliest master of all). Employees. I was a slave to everyone for so many years. And the more shackles I had on, the lonelier I got
Much of the time, even when I had those moments of success, I didn't know how to turn it into a better life. I felt ugly and then later, I felt stupid when I would let the success dribble away down the sink
I love writing because every now and then that ugliness turns into honesty. When I write, I'm only a slave to myself. When I do all of those other things you ask about, I'm a slave to everyone else
Some links
33 Unusual Tips to Being a Better Write
"The Tooth
(one of my favorite posts on my blog
2. What inspires you to get up and start working/writing every day
The other day I had breakfast with a fascinating guy who had just sold a piece of his fund of funds. He told me what "fracking" was and how the US was going to be a major oil player again. We spoke for two hours about a wide range of topics, including what happens when we can finally implant a google chip in our brains
After that I had to go onto NPR because I firmly believe that in one important respect we are degenerating as a country - we are graduating a generation of indentured servants who will spend 50 years or more paying down their student debt rather than starting companies and curing cancer. So maybe I made a difference
Then I had lunch with a guy I hadn't seen in ten years. In those ten years he had gone to jail and now I was finally taking the time to forgive him for something he never did to me. I felt bad I hadn't helped him when he was at his low point. Then I came home and watched my kid play clarinet at her school. Then I read until I fell asleep. Today I did nothing but write. Both days inspired me
It also inspires me that I'm being asked these questions. Whenever anyone asks me to do anything I'm infinitely grateful. Why me? I feel lucky. I like it when someone cares what I think. I'll write and do things as long as anyone cares. I honestly probably wouldn't write if nobody cared. I don't have enough humility for that, I'm ashamed to admit
3. Your new book "How to be the luckiest person alive" has just come out. What is it about
When I was a kid I thought I needed certain things: a college education from a great school, a great home, a lot of money, someone who would love me with ease. I wanted people to think I was smart. I wanted people to think I was even special. And as I grew older more and more goals got added to the list: a high chess rating, a published book, perfect weather, good friends, respect in various fields, etc. I lied to myself that I needed these things to be happy. The world was going to work hard to give me these things, I thought. But it turned out the world owed me no favors
And gradually, over time, I lost everything I had ever gained. Several times. I've paced at night so many times wondering what the hell was I going to do next or trying not to care. The book is about regaining your sanity, regaining your happiness, finding luck in all the little pockets of life that people forget about. It's about turning away from the religion you've been hypnotized into believing into the religion you can find inside yourself every moment of the day
[Note: in a few days I'm going to do a post on self-publishing and also how to get the ebook for free. The link above is to the paperback. Kindle should be ready soon also.
Related link: Why I Write Books Even Though I've Lost Money On Every Book I've Ever Writte
4. Is it possible to accelerate success? If yes, how
Yes, and it's the only way I know actually to achieve success. It's by following the Daily Practice I outline in this post:
It's the only way I know to exercise every muscle from the inside of you to the outside of you. I firmly believe that happiness starts with that practice
5. You say that discipline, persistence and psychology are important if one has to achieve success. How can one work on improving "psychology" part
Success doesn't really mean anything. People want to be happy in a harsh and unforgiving world. It's very difficult. We're so lucky most of us live in countries without major wars. Our kids aren't getting killed by random gunfire. We all have cell phones. We all can communicate with each other on the Internet. We have Google to catalog every piece of information in history! We are so amazingly lucky already
How can it be I was so lucky to be born into such a body? In New York City of all places? Just by being born in such a way on this planet was an amazing success
So what else is there? The fact is that most of us, including me, have a hard time being happy with such ready-made success. We quickly adapt and want so much more out of life. It's not wars or disease that kill us. It's the minor inconveniences that add up in life. It's the times we feel slighted or betrayed. Or even slightly betrayed. Or overcharged. Or we miss a train. Or it's raining today. Or the dishwasher doesn't work. Or the supermarket doesn't have the food we like. We forget how good the snow tasted when we were kids. Now we want gourmet food at every meal
Taking a step back, doing the Daily Practice I outline in the question above. For me, the results of that bring me happiness. That's success. Today. And hopefully tomorrow
6. You advocate not sending kids to college. What if kids grow up and then blame their parents about not letting them get a college education
I went to one of my kid's music recitals yesterday. She was happy to see me. I hugged her afterwards. She played "the star wars theme" on the clarinet. I wish I could've played that for my parents. My other daughter has a dance recital in a few weeks. I tried to give her tips but she laughed at me. I was quite the breakdancer in my youth. The nerdiest breakdancer on the planet. I want to be present for them. To love them. To let them always know that in their own dark moments, they know I will listen to them. I love them. Even when they cry and don't always agree with me. Even when they laugh at me because sometimes I act like a clown
Later, if they want to blame me for anything at all then I will still love them. That's my "what if"
Two posts
I want my daughters to be lesbian
Advice I want to give my daughter


7. Four of your favorite posts from The Altucher Confidential
As soon as I publish a post I get scared to death. Is it good? Will people re-tweet? Will one part of the audience of this blog like it at the expense of another part of the audience. Will I get Facebook Likes? I have to stop clinging to these things but you also need to respect the audience. I don't know. It's a little bit confusing to me. I don't have the confidence of a real writer yet
Here are four of my favorites
How I screwed Yasser Arafat out of $2mm (and lost another $100mm in the process
It's Your Fault
I'm Guilty of Torturing Wome
The Girl Whose Name Was a Curs
Although these three are favorites I really don't post anything unless it's my favorite of that moment
8. 3 must-read books for aspiring entrepreneurs
The key in an entrepreneur book: you want to learn business. You want to learn how to honestly communicate with your customers. You want to stand out
The Essays of Warren Buffett by Lawrence Cunningha
"The Thank you Economy" by Gary Vaynerchu
"Purple cow" by Seth Godi
9. I love your writing, so do so many others out there. Who are your favorite writers
"Jesus's Son" by Denis Johnson is the best collection of short stories ever written. I'm afraid I really don't like his novels though
"Tangents" by M. Prado. A beautiful series of graphic stories about relationships
Other writers: Miranda July, Ariel Leve, Mary Gaitskill, Charles Bukowski, Celine Bags Outlet, Sam Lipsyte, William Vollmann, Raymond Carver. Arthur Nersesian. Stephen Dubner
Many writers are only really good storytellers. Most writers come out of a cardboard factory MFA system and lack a real voice. A real voice is where every word exposes ten levels of hypocrisy in the world and brings us all the way back to see reality. The writers above have their own voices, their own pains, and their unique ways of expressing those pains. Some of them are funny. Some a little more dark. I wish I could write 1/10 as good as any of them

10. You are a prolific writer. Do you have any hacks that help you write a lot in little time
Coffee, plus everything else coffee does for you first thing in the morning
Only write about things you either love or hate. But if you hate something, try to find a tiny gem buried in the bag of dirt so you can reach in when nobody is looking and put that gem in your pocket. Stealing a diamond in all the shit around us and then giving it away for free via writing is a nice little hack, Being fearless precisely when you are most scared is the best hack

11. I totally get and love your idea about bleeding as a writer, appreciate if you share more with the readers of this blog
Most people worry about what other people think of them. Most people worry about their health. Most people are at a crossroads and don't know how to take the next step and which road to take it on. Everyone is in a perpetual state of 'where do I put my foot next'. Nobody, including me, can avoid that
You and I both need to wash our faces in the morning, brush our teeth, shower, shit, eat, fight the weather, fight the colds that want to attack us if we're not ready. Fight loneliness or learn how to love and appreciate the people who want to love you back. And learn how to forgive and love the people who are even more stupid and cruel than we are. We're afraid to tell each other these things because they are all both disgusting and true
You and I both have the same color blood. If I cut my wrist open you can see the color of my blood. You look at it and see that it's the same color as yours. We have something in common. It doesn't have to be shameful. It's just red. Now we're friends. No matter whom you are or where you are from. I didn't have to lie to you to get you to be my friend
Related Links
How to be a Psychic in Ten Easy Lesson
My New Year's Resolution in 199
12. What is your advice for young entrepreneurs
Only build something you really want to use yourself. There's got to be one thing you are completely desperate for and no matter where you look you can't find it. Nobody has invented it yet. So there you go - you invent it. If there's other people like you, you have a business. Else. You fail. Then do it again. Until it works. One day it will
Follow these 100 Rules
The 100 Rules for Being a Good Entrepreneur
And, in particular this
The Easiest Way to Succeed as an Entrepreneu
In my just released book I have more chapters on my experiences as an entrepreneur
13. I advocate the concept of working at a job while building your business. You have of course lived it. Now as you look back, what is your take on this? Is it possible to make it work while sailing on two boats

Your boss wants everything out of you. He wants you to work 80 hours a week. He wants to look good taking credit for your work. He wants your infinite loyalty. So you need something back
Exploit your employer. It's the best way to get good experience, clients, contacts. It's a legal way to steal. It's a fast way to be an entrepreneur because you see what large companies with infinite money are willing to pay for. If you can provide that, you make millions. It's how many great businesses have started and will always start. It's how every exit I've had started

14. Who is a "person with true moral fiber"? In current times are there any role models who are people with true moral fiber
I don't really know the answer. I think I know a few people like that. I hope I'm someone like that. And I pray to god the people I'm invested in are like that and my family is like that
I find most people to be largely mean and stupid, a vile combination. It's not that I'm pessimistic or cynical. I'm very much an optimist. It's just reality. Open the newspaper or turn on the TV and watch these people
Moral fiber atrophies more quickly than any muscle on the body. An exercise I do every morning is to promise myself that "I'm going to save a life today" and then leave it in the hands of the Universe to direct me how I can best do that. Through that little exercise plus the Daily Practice described above I hope to keep regenerating that fiber
15. Your message to the readers of this blog
Skip dinner. But follow me on Twitter.
Read more posts on The Altucher Confidential � More from The Altucher Confidentia
Life is Like a Game. Here�s How You Master ANY Gam

Step By Step Guide to Make $10 Million And Then Totally Blow

Can You Do One Page a Day?

Retrieved from "https://en.formulasearchengine.com/index.php?title=Kinetic_width&oldid=270323"