After doing some testing with the Elmer's glue the other night and seeing that it held up OK, I tried another quick test using a smaller chunk of foam and again, using Elmer's. But this time I only ran three thin strips of glue instead of spreading out a patch of it.
My thoughts were that the patch was still showing some wet glue in the middle of the patch. As the glue dried around the edges it sealed the inner bit of glue so it wouldn't dry. I've seen this happen when others tried using Elmer's or Titebond to glue the rigid foam board together. By running strips of glue, I figured the air could get between the strips and being thin strips, would stand a better chance of drying completely.
After letting the strip test dry overnight, it took quite a bit of pressure to break the chunk lose. I actually started to dent the soft foam with my fingers while trying to dislodge it. So I think strips of Elmer's will do the trick.
Tonight I got the first two sheets glued down.