Travis:
Your problem is clear: you are using a container for Rectangles to store Line Segments.
Rectangles happen to have certain characteristics, ie, they are normal. The left coordinate is always less than the right one, and similarly for top and bottom.
Line segments don't have such constraints.
You need to transform the 2 endpoints in 2 separate operations.