If you set it up properly, all compilation can be done on you main PC(s), with no need to have an entire build-chain on the Pandora.
You can tell portage to install to another root/install with the ROOT=... -variable. Gentoo has split up build-time and run-time dependencies, build-time deps will be installed in the current install, while run-time dependencies will be installed to wherever ROOT points to. You also have a PORTAGE_CONFIGROOT -variable, so that you can keep the portage-config for the target-system and your build-system separate.
You also have the emerge-option "--buildpkg (-

" to build a binary package of whatever you build, and you have the options "--usepkgonly (-K)", "--usepkg (-k)", "--getbinpkg (-g)", and "--getbinpkgonly (-G)" to install those packages(respectively: only use bin-packages, allow non-bin-packages, get bin-packages from a server, and get them from a server exclusively). Build-time-only dependencies won't be included for installing binary packages.
Default emerge-options can be set with the If you set it up properly, all compilation can be done on you main PC(s), with no need to have an entire build-chain on the Pandora.
You can tell portage to install to another root/install with the ROOT=... -variable. Gentoo has split up build-time and run-time dependencies, build-time deps will be installed in the current install, while run-time dependencies will be installed to wherever ROOT points to. You also have a PORTAGE_CONFIGROOT -variable, so that you can keep the portage-config for the target-system and your build-system separate.
You also have the emerge-option "--buildpkg (-

" to build a binary package of whatever you build, and you have the options "--usepkgonly (-K)", "--usepkg (-k)", "--getbinpkg (-g)", and "--getbinpkgonly (-G)" to install those packages(respectively: only use bin-packages, allow non-bin-packages, get bin-packages from a server, and get them from a server exclusively). Build-time-only dependencies won't be included for installing binary packages.
Default emerge-options can be set with the "EMERGE_DEFAULT_OPTS"-option which can be set in make.conf.
See the man-pages of emerge, portage, and make.conf for more info.
So, you can make a gentoo-cross-compile-toolchain(preferably in a chroot, makes it easy to distribute as well), that compiles everything to a separate location, and makes binary packages from it, and install those binary packages on the pandora. You won't need gcc etc. on the pandora, and, with the "-K"-option, you won't even need a portage-tree, which takes up several 100MB, or ~50MB compressed.
Going further, you can then create a preconfigured image for the pandora, that uses a repository full of those binary packages. Users that want to roll their own packages can do so as well. Gentoo's repository is huge, and as such, this makes it easy to provide a large amount of packages.
You have basically described our entire system. Lets me know we are on the right track. Are you interested in doing some testing?