#### Motivation

Let $\Omega=\{\omega_{1}, \omega_{2},\omega_{3} \}$ be a space space and $P(\{\omega_{i}\}=\frac{1}{3}, \forall i=1,2,3$. Let $X:\Omega \rightarrow \mathbb{R}$ and $Y:\Omega \rightarrow \mathbb{R}$ be discrete random variables defined with:

$$X(\omega_{1})=1, X(\omega_{2})=2, X(\omega_{3})=3$$ $$Y(\omega_{1})=2, X(\omega_{2})=3, X(\omega_{3})=1$$ $X$ and $Y$ has the same distribution:

Lets take a look at the distribution of $X+Y$. We know that $(X+Y)(\omega_{1})=X(\omega_{1})+Y(\omega_{1})=1+2=3$. Similarly, for $(X+Y)(\omega_{2})=4$ and $(X+Y)(\omega_{3})=5$. The distribution looks like this:

Naturally we would think the distribution of $X+X$ looks the same as $X+Y$ since $X \sim Y$. However, that’s not the case.

Generally, function $g:\mathbb{R}^{2} \rightarrow \mathbb{R}$, $g(X,Y)$ is random variable. However, for finding its distribution it’s not enough to know the the distribution of $X$ and $Y$. We need to find their combined distribution, i.e. the distribution of random vector $(X,Y)$.

#### Definition

For discrete random variables $X$ and $Y$ we define discrete random vector as a function $(X,Y): \Omega \rightarrow \mathbb{R}^{2}$.

Let $X$ have values $a_{i}$, $\forall i \in \mathbb{N}$ and $Y$ values $b_{j}$, $\forall j \in \mathbb{N}$. Pairs $(a_{i}, b_{j})$ and their probabilities $$p_{i,j}:=P(X=a_{i}, Y=b_{j})$$ are the distribution of discrete random vector $(X,Y)$.

The table of distribution looks like this:

For example,

$$q_{1}=\sum_{j=1}^{\infty} p_{1,j} = \sum_{j=1}^{\infty} P(X=a_{i}, Y=b_{j}) =P(\{X=a_{1}\} \cap (\cup_{j=1}^{\infty}\{Y=b_{j}\}))=P(X=a_{1})$$

because $(\cup_{j=1}^{\infty}\{Y=b_{j}\})=\Omega$.

From the table we can easily see that

**Proposition.**

**(i)** $X$ and $Y$ are independent if and only if $$p_{i,j}=P(X=a_{i}, Y=b_{j})=P(X=a_{i}) P( Y=b_{j})=q_{i} r_{j}, \forall i,j \in \mathbb{N}$$

**(ii)** If $g:\mathbb{R}^{2} \rightarrow \mathbb{R}$ then for every $c \in \mathbb{R}$

$$\displaystyle{P(g(X,Y)=c)= \sum^{\infty} _{\substack {i,j=1\\ g(a_{i},b_{j})=c}} P(X=a_{i}, Y=b_{j}) }$$ Consequently, $$E[g(X,Y)] =\sum_{i,j=1}^{\infty} g(a_{i}, b_{j}) p_{i,j}$$

#### Covariance

Covariance is a measure of the strength of the correlation between two or more random variables. It shows how much the variables change together. In other words, it’s a measure of the joint variability of random variables.

**Definition.** Let $X$ and $Y$ be random variables and $Var(X), Var(Y) < \infty$. **Covariance** of $X$ and $Y$ is $$Cov(X,Y):=E[(X-E[X])(X-E[Y])]=E[XY]-E[X] E[Y]$$ If $Cov(X,Y)=0$ variables $X$ and $Y$ aren’t correlated, otherwise they are correlated.

Sometimes it’s difficult to estimate the strength of correlation solely on covariance. For example, does $Cov(X,Y)=65$ mean $X$ and $Y$ are strongly correlated or not?

For that reason we define **correlation coefficient** $$\rho (X,Y)= \frac{Cov(X,Y)}{\sqrt{Var(X)} \sqrt{Var(Y)}} \in [-1,1]$$

We have three possibilities:

**positive correlation**, $\rho \in (0,1] \rightarrow$ both variables change into the same direction**neutral correlation**, $\rho=0 \rightarrow$ no relationship between variables**negative correlation**, $\rho \in [-1,0) \rightarrow$ variables change in the opposite direction

Intuitively, if $\rho (X,Y)$ is close to $-1$ or $1$ that indicates strong dependence between $X$ and $Y$. Analogously, if $X$ and $Y$ are independent, then $\rho (X,Y)=0$, i.e. they aren’t correlated. However, if $\rho (X,Y)=0$ that doesn’t indicate that $X$ and $Y$ are independent.

**Example.** Let $X$ be random variable with distribution and $Y=X^{2}$. In this case $X$ and $Y$ are dependent, however $$Cov(X,Y)=E[XY]-E[X][Y]=E[X^{3}]- 0 \cdot E[Y]=0$$ Consequently, $X$ and $Y$ aren’t correlated.

In conclusion, $Cov(X,Y)=0$ generally doesn’t imply that $X$ and $Y$ are independent. That’s because covariance measures only one form of dependence, i.e. “linear” dependence.

#### Example

Lets throw two symmetric dice. Let $X$ be the number of $6$s we get, and $Y$ the number of $1$s. Find:

**(a)** Distribution of random vector $(X,Y)$ and random variables $X$ and $Y$. Are they independent?

**(b)** $\rho (X,Y)$

**(c****)** Distribution of $X+2Y$.

**Solution**

**(a)** Distribution of $(X,Y)$

Since we only have $2$ dice the number of $6$s and $1$s can only be $0, 1$ or $2$. Lets start from beginning.

Lets say we got zero $6$s and $1$ss. That means both dice have $4$ options left. Since they are independent, we use the multiplication rule. $$\displaystyle{P(X=0, Y=0)= \frac{4^{2}}{6^{2}}=\frac{16}{36}}$$

Furthermore, lets say we have zero $6$s and one $1$. That means one die has only one option $(1)$ while the other has $4$ – there can’t be $1$ or $6$. If $1$ rolled on the second die too, it would mean we had two $1$ and not only one. Moreover, we don’t know which die had rolled $1$, so we can have two options. Consequently, $$\displaystyle{P(X=0, Y=1)= 2 \cdot \frac{4}{6^{2}} =\frac{8}{36}}$$

Similarly for $\displaystyle{(X=1, Y=0)=\frac{8}{36}}$. We use the same way of thinking, we just switch the numbers.

Additionally, for one $1$ and one $6$ we only have an option when one die has $1$ and another has $6$. However, since we can switch dice, we have $2$ options. $$\displaystyle{P(X=1, Y=1)= 2 \cdot \frac{1}{6^{2}} =\frac{2}{36}}$$

For situation where we have $$\displaystyle{(X=0, Y=2)= \frac{1}{6^{2}}= \frac{}{36}}$$

Options where:

$(X=1, Y=2)$, $(X=2, Y=1)$ or $(X=2, Y=2)$ cannot happen because we only have 2 dice. Consequently, we cannot have rolled two $6$s and one $1$.

Finally, lets make a table of distribution

The distribution of $X$ and $Y$ separately will be symmetric.

**Are $X$ and $Y$ independent? **

Random variables $X$ and $Y$ are independent if $$P(X=a_{i}, Y=b_{j})=P(X=a_{i}) \cdot P(Y=b_{j})$$ The best way to see if two variables are independent while using the distribution of random vector is to look for $0$s.

Lets see if $P(X=0, Y=0)=P(X=0) \cdot P(Y=0)$. Reading from the table we have:

$$P(X=0, Y=0)=\displaystyle{\frac{16}{36} \neq \frac{25}{36} \cdot \frac{25}{36} = P(X=0) \cdot P(Y=0)}$$

As a result, $X$ and $Y$ aren’t independent.

**(b)** $\rho (X,Y) = ?$

For starters lets look at the formula of correlation coefficient

$$\rho (X,Y)= \frac{Cov(X,Y)}{\sqrt{Var(X)} \sqrt{Var(Y)}}$$

Firstly, lets find $Cov(X,Y)$. $$Cov(X,Y)=E[XY]-E[X]E[Y]$$

$\displaystyle{E[X]=0 \cdot \frac{25}{36} + 1 \cdot \frac{10}{36} + 2 \cdot \frac{1}{36}=\frac{1}{3}}$

Similarly, $\displaystyle{E[Y]= \frac{1}{3}}$.

For $E[XY]$ we have to multiply every separate value of both $X$ and $Y$ and their combined probability. For starters:

$$X=0, Y=0, P= \frac{16}{36} \rightarrow 0 \cdot 0 \cdot \frac{16}{36}$$

We then sum up all the products:

$$\displaystyle{E[XY] = 0 \cdot 0 \cdot \frac{16}{36} + 0 \cdot 1 \cdot \frac{8}{36} + 0 \cdot 2 \cdot \frac{1}{36} + 1 \cdot 0 \cdot \frac{8}{36} + … 1\cdot 1 \cdot \frac{2}{36} +…}$$

Since we have $0$ in every product besides one, $\displaystyle{E[XY]=\frac{2}{36}= \frac{1}{18}}$

Consequently, $\displaystyle{Cov(X,Y)=\frac{1}{18}-\frac{1}{9}=-\frac{1}{18}}$

Secondly, we need variance of each random variable.

$$Vax(X)= Var(Y)= E[X^{2}]-(E[X])^{2}=\frac{7}{18} -(\frac{1}{3})^{2}=\frac{5}{18}$$

$$\rho (X,Y)= \frac{\frac{-1}{18}}{\sqrt{\frac{5}{18} \frac{5}{18}}=\frac{-1}{5}}=-0.2$$

This is another way to see if variables are independent. $\rho \neq 0$, hence $X$ and $Y$ are dependent.

**(c)** Distribution of $X+2Y$

Possible values of $X$ and $Y$ are still $0$, $1$ or $2$. However, we need the values of $X+2Y$. Lets make another table. Be careful, this is not a distribution table. It just helps us to find the right values. Even though we are finding the distribution of $X+2Y$ we still have to take into consideration that $X$ and $Y$ represent the number of $1$s and $6$s. As a result, we can’t have outcomes like before, i.e. $X=2, Y=1$.

We fill in the table in the following the $X+2Y$:

$\bullet$ $X=0, Y=0 \rightarrow X+2Y=0$

$\bullet$ $X=0, Y=1 \rightarrow X+2Y=2$

… and so on. The table looks like this.

We can see that the values that appear are $0$, $1$, $2$ $3$ and $4$. Consequently, those are the possible values for $X+2Y$.

Lets find their probabilities.

For $X+2Y=0$ there is only one possible solution $\rightarrow (X=0, Y=0)$. From the table of distribution (in part **(a)**) we read the probability of that outcome which is $\displaystyle{\frac{16}{36}=\frac{4}{9}}$.

For $X+2Y=1$ there is only one possible solution $\rightarrow (X=1, Y=0)$. It’s probability is $\displaystyle{\frac{8}{36}=\frac{2}{9}}$.

For $X+2Y=2$ we have multiple solutions

$\bullet$ $\displaystyle{(X=0, Y=1) \rightarrow P= \frac{2}{9}}$

$\bullet$ $\displaystyle{(X=2, Y=0) \rightarrow P=\frac{1}{36}}$

Those are mutually exclusive outcomes, so we sum up their probabilities. $\displaystyle{P=\frac{1}{4}}$

For $X+2Y=3$ there is only one possible solution $\rightarrow (X=1, Y=1)$. It’s probability is $\displaystyle{\frac{2}{36}=\frac{1}{18}}$.

For $X+2Y=4$ we have multiple solutions

$\bullet$ $(X=2, Y=1) \rightarrow P= 0$

$\bullet$ $\displaystyle{(X=0, Y=2) \rightarrow P=\frac{1}{36}}$

Combined probability is $\displaystyle{\frac{1}{36}}$.

Finally, the distribution of $X+2Y$ looks like this