What do you mean Photoshop Reverse Engineering? If you mean "has anyone ever looked at Photoshop by reverse engineering part of it?", yes for sure. I'm sure there are a few articles that focus on a specific part of photoshop, but if you mean projects I do not think so. There are plenty of open-source projects that are based on Photoshop UX, however no one has ever tried to reverse engineering Photoshop and then integrate into an open-source project.

I think reverse engineering Photoshop is somehow useless for some parts, except for curiosity: some algorithms that it uses are well-known and are based on infinite amount of literature of CS (e.g. image processing and elaboration). For sure, there are some others algorithms that are more obscure/secret that you can might explore for your own. In addition, think also about the problems you might have re-implementing an algorithm. Adobe laywers are pretty powerful haha.