All You Need Is a Net