The CR2C2 team at UTK has developed a novel Perception Error Model (PEM) using a modified Fully Embedded Time-Series Generative Adversarial Network (FETSGAN). Trained on the Comma2k19 dataset, the PEM generates realistic perception error sequences by modeling the probabilistic behavior of vision-based sensors, particularly as implemented in the Supercombo perception stack of OpenPilot. It is shown that these errors are not just stochastic noise but are informed by observed patterns in real-world perception system failures, allowing for the injection of plausible inaccuracies into simulation experiments. This approach bridges the gap between high-fidelity simulations and real-world sensor limitations, enabling more robust validation of safety-critical control systems such as Model Predictive Controllers (MPCs).

Link to the tool: GitHub - jbeck9/FETSGAN at pem_testing 

Related Publications: 

• Beck, J., Chakraborty, S. Fully embedded time series generative adversarial networks. Neural Comput & Applic 36, 14885–14894 (2024). https://doi.org/10.1007/s00521-024-09825-5

• Beck, J., Huff, S., & Chakraborty, S. (2024). Diagnosing and Predicting Autonomous Vehicle Operational Safety Using Multiple Simulation Modalities and a Virtual Environment. arXiv preprint arXiv:2405.07981.