Make logs rotate based on size or time
As Spark error and warn logs tend to grow quickly, they could take a lot of space and it could be hard to open such logs when it has grown a lot. So it would be useful to have rotating logs. Based either on size (say new log file should be created when current reaches 1 MB or similar size). There should also be a limit of files, so it won't create thousands of files. 10 seems enough. Or there could be new log file for each month and Spark could save only 5 of them and delete older months.