\fbox `\fbox` kann nur horizontales Material. Du musst also z.B. eine \parbox `\parbox` dazwischen schieben:
\documentclass[10pt,a4paper]{article}
\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{subfigure}
\usepackage{caption}
\usepackage{graphicx}
\begin{document}
\begin{figure}[h]
\centering
\fbox{%
\parbox{0.49\textwidth}{%
\subfigure[XX]{\includegraphics[width=0.49\textwidth]{example-image}}
\subfigure[YY]{\includegraphics[width=0.49\textwidth]{example-image}}}}
\caption{test}
\label{}
\end{figure}
\end{document}
