I have used silicon brake fluid twice: In a 1973 BMW 3.0CS that I restored and my boat trailer. Neither have showed any adverse effects. I kept the 3.0CS for 10+ years after restoration and never changed the fluid. The brakes worked great (no spongy pedal) until the day I sold the car. I placed a big "STOP: use only silicon brake fluid" label on the brake booster next to the fluid reservoir.
The boat trailer brakes are still working fine as well.
I am debating internally on whether to fill my new (power front disk) brake system with silicon or conventional.