One of the big announcements at Mozilla Summit 2013, despite Firefox OS being the focus of the event, was their research (with Samsung) into a new rendering engine, Servo. Rendering HTML5 is horrifically complex, so creating a new rendering engine from scratch is a big “nope!” for basically all organizations. Mozilla saw this as a big potential, because current engines are very difficult to scale to multiple cores, so they went in to this as a no-assumptions experiment.
At the time, they didn't know whether Servo would be built up into a full rendering engine, or whether it would be picked apart and pulled back into their current engine, Gecko. Mozilla has now unveiled Quantum, and the first sentence of its MozillaWiki entry is “Quantum is not a new web browser.” They go on to say that they will be “building on the Gecko engine as a solid foundation”. So it seems pretty clear that, like they've recently done with their media file parser in Firefox 48.
While this will likely not have the major impact that “boom, new engine” would, in terms of performance, this piece-wise method should be quicker than bulking up Servo. Mozilla expects that big changes will begin to land next year.