1. State the problem: The binary operation $*$ is defined on the set $\mathbb{R}$ by $a * b = \frac{ab}{4}$ and we are to find $\sqrt{2} * \sqrt{6}$.\n2. Formula and rules: Use the definition $a * b = \frac{ab}{4}$ for all real $a,b$.\n3. Important rules: Multiply radicals using $\sqrt{m}\sqrt{n}=\sqrt{mn}$ and simplify radicals by factoring perfect squares.\n4. Substitute values: Substitute $a=\sqrt{2}$ and $b=\sqrt{6}$ to get $$\sqrt{2} * \sqrt{6} = \frac{\sqrt{2}\sqrt{6}}{4}.$$\n5. Simplify the product of radicals: Using the rule, $$\frac{\sqrt{2}\sqrt{6}}{4}=\frac{\sqrt{12}}{4}.$$\n6. Simplify the radical: Since $\sqrt{12}=\sqrt{4\cdot3}=2\sqrt{3}$ we have $$\frac{\sqrt{12}}{4}=\frac{2\sqrt{3}}{4}.$$\n7. Cancel common factor: Show cancellation explicitly: $$\frac{2\sqrt{3}}{4}=\frac{\cancel{2}\sqrt{3}}{\cancel{4}}=\frac{\sqrt{3}}{2}.$$\n8. Final answer: Therefore $\sqrt{2} * \sqrt{6}=\frac{\sqrt{3}}{2}$, which is option C.\n