When the right time to build a design system is a question a lot of small teams are asking these days. If you talk to different product teams the opinions can vary.
In my own time thinking about this for AVVAY I've surfaced a few thoughts that our team will be keeping top of mind.
I believe the most important principle when thinking through this question is whether or not the team is clear on why they need a design system. For a lot of small or fast moving teams It may not make much sense to invest the time and energy into building a system. The catalyst should first and foremost be to give the designer the ability to export their knowledge. Giving broader access to this knowledge is what will give future team members the ability to take what you contribute and build something new.
Another factor in the decision making process should be whether or not outside stakeholders need to have a fundamental understanding of the structure of the product. Giving transparency into what the team is building and also how they're building it will be important part of the growth of the team and product. Also it should be understood that a good design system will be a living document and will need ongoing support. Documentation will only be the first step.
Lastly, your team's goal should be to create something that lives independently from the source that created it. A critical question should be "What do we need to capture so that any team member can build with all the understanding they need?" This is our goal at AVVAY especially when thinking about scaling issues. If a future designer doesn't have access to the rest of the team their understanding should come from reading guidelines and examples we've set forth and getting a really good sense of how we've realized the product.
So again, the answer of when may not be as important as figuring the why of the question. If there is a clear understanding of the goals and motives you'll have a more usable and valuable design system the whole team can use and build upon.