There is no easy answer to your predicament. Most likely, you should separate, clearly explaining to your partner that you have been disobedient to God in living together before marriage, and that to continue to cohabit constitutes continued, willful fornication. Scripture is clear that sexual immorality is a sin and that our bodies are designed to glorify the Lord: “You were bought at a price. Therefore honor God with your bodies” (1 Corinthians 6:20). God did not save you in order for you to continue in sin (Romans 6:1–4).
You could marry, obviating the concerns of immorality, but, if you do, you form an unequal yoke that God forbids (2 Corinthians 6:14). Now that Christ has entered your life, your primary love is for Him (Luke 14:26). And loving the Lord goes hand in hand with obeying the Lord (John 14:15).
The biblical principles presented so far are straightforward; however, there may be other considerations that complicate the issue. For example, are there children involved? Also, in the state where you live, what are the laws concerning common law marriage? It could be that you are already legally married, depending on the details of the marriage law in your locale. If you did exchange informal vows not in the presence of an officiant, then one possibility is to solemnize the marriage to which you are already committed. If your unbelieving marriage partner does not wish to ratify the relationship with an official license, then the principle of 1 Corinthians 7:15 might apply.
In addition to praying about this issue and searching the Scriptures, it is imperative to find a godly counselor—a pastor or someone who truly understands the Bible and who has a lot of life experience—with whom you can share all the details of your situation. As you follow the precepts of Scripture, pray for boldness and grace. The way to go will be made clear. Believe that God really knows what is best for you. Trust that He has your best interests at heart.