Superconducting onset Tc ~ 39 K below the magnetic ordering temperature Tm ~ 49 K is observed in the new magnetic superconductor RuCa2PrCu2O8+δ. This compound is crystallized with the orthorhombic distortion of the tetragonal RuSr2GdCu2O8+δ-type structure (space group P4/mmm, Tm ~ 136 K, Tc(max) ~ 65 K). With similar RuO6 octahedral and CuO5 pyramidal units, superconductivity from the Cu-O bi-layers is apparently coexisting with the very weak ferromagnetic order. Low temperature anomalous magnetic properties observed in both field-cooled (FC) and zero-field-cooled (ZFC) modes are closely related to the complex interplay and coexistence between weak ferromagnetic order and superconductivity as well as from structural distortion and sensitive sample oxygen concentration variation. On the other hand, the tetragonal RuSr2PrCu2O8+δ compound is non-superconducting and shows spin-glass-like behavior at low temperature.