I suspect Epiphany got thrown in connection with Christmas because of what Epiphany is really about. As the name implies, it's about revelation. It's about God's manifestation in the world.
The Nativity is a huge part of that manifestation. So is the star that the Wise Men saw and followed, and so are the other events we celebrate, such as Jesus' Baptism on the first Sunday in the Season after the Epiphany.
During this season, remember that is about so much more than just the Wise Men. It's so much more than just an "ordinary time" of the church. This season is a true extension of Christmas. This season is about how God was made manifest in the world through Jesus, and that all starts with God's birth into this world as a tiny child.
Further Reading:
Alexander, Neil. Creating Liturgical Time. New York: Church Publishing, 2014.