We want a different policy,” co-leader Alexander Gauland said following the historic surge.
Chancellor Angela Merkel has been re-elected for a fourth term but her conservative CDU/CSU bloc received its worst result in almost 70 years.
Mrs Merkel is beginning negotiations to form a new coalition government.
Her bloc’s current coalition partners, the Social Democrats (SPD), say they will go into opposition after historic losses.
The AfD’s campaign capitalised on a backlash over Mrs Merkel’s decision to open Germany’s borders to migrants and refugees in 2015.
Its success has shocked Germany’s political establishment, and protests against the anti-Islam party have been held in several cities following the election result.
One million people, foreigners, being brought into this country are taking away a piece of this country and we as AfD don’t want that,” Mr Gauland told a news conference on Monday.
“We say I don’t want to lose Germany to an invasion of foreigners from a different culture. Very simple.”
Why this vote is a turning point
What does Alternative for Germany (AfD) want?
He said the AFD – which is expected to take 94 seats in the 709-seat federal parliament as the third-largest party – had been elected “to uncompromisingly address” immigration issues.
But discord is already evident in the party – with another leader, Frauke Petry, declaring at the same news conference she would not join the AfD’s parliamentary group despite winning a seat, before abruptly leaving the room to the apparent surprise of other leaders
Ms Petry, who is the best-known AfD figure, said there was “disagreement over content” in the party and had earlier criticised Mr Gauland for saying the AfD would “hound” Mrs Merkel.
What does the result mean for Mrs Merkel?
While her bloc remains the largest in the Bundestag, the lower house of parliament, it is the worst result for the alliance between the Christian Democrats (CDU) and the Christian Social Union (CSU) since 1949, when national elections were held in Germany for the first time after World War Two.
Addressing supporters, Mrs Merkel, who has been in the job for 12 years, said she had hoped for a “better result”
Mrs Merkel also said her government would have to deal with economic and security issues as well as addressing the root causes of migration – one of the main reasons behind the AfD’s result.
“Today we can say that we now have a mandate to assume responsibility and we’re going to assume this responsibility calmly, talking with our partners of course.”
A chaotic day for German politics: Jenny Hill, BBC News, Berlin
It has been a long and bruising election campaign. Angela Merkel may have won the election but it does not feel like much of a victory.
This election will go down in the history books for two reasons. Mrs Merkel may have won a fourth term but it is her worst-ever general election result. And right-wing nationalists are now part of the German establishment.
The result is a verdict, perhaps, on Mrs Merkel’s decision to open Germany’s doors to one million refugees.
What is the political norm in many other European countries was considered unthinkable in post-war Germany. Not any more.
Read more from Jenny
What are her coalition options?
The SPD had their worst election result in the post-war era. The party’s loss of support while junior partner in government saw leader Martin Schulz declare on Sunday evening the end of the “grand coalition” with Mrs Merkel’s alliance, to cheers and applause.
He recognised it was a “bitter day” for the Social Democrats but vowed to prevent the AfD from being the main opposition party and enjoying certain privileges as a result.
The SPD was given “a mandate to be a strong opposition in this country; a mandate to defend democracy in this country against all of those who question and attack it”, Mr Schulz told supporters on Saturday
With the possibility of an alliance with the SPD rejected, Mrs Merkel’s options are narrow, and the process of forming a new coalition could take months.
Six parties will be in the German parliament for the first time since the 1950s.
Election marks unprecedented shift
Long read: Angela Merkel’s quiet power
The most likely scenario is of a “Jamaica” coalition, so-called because of the colours of Jamaica’s flag. It includes the black CDU/CSU, the yellow, business-friendly Free Democrats (FDP) – who are returning to parliament after a four-year hiatus – and the Greens.
It is not a marriage made in heaven, as the Greens want to phase out 20 coal-fired power plants and the FDP disagree, but it is the only formation that would guarantee enough seats in the new Bundestag, German broadcaster ZDF says.
All parties have rejected working with the AfD.
Why did the AfD gain support?
Alternative for Germany was founded in 2013 as an anti-euro party but later turned its focus to immigration and Islam.
It called for a ban on minarets and declared Islam incompatible with German culture. Several of its candidates have been linked to far-right remarks.
Those hardline positions helped it to win seats in 13 of Germany’s 16 state parliaments in the last few years.
Its election campaign posters carried messages such as “Stop Islamicisation. Vote AFD” alongside images of women wearing burkas, and “Burkas? We stand for bikini
It performed particularly well in what was formerly East Germany, taking 21.5% of the vote as the second most popular party.
In Saxony, the AfD narrowly beat the CDU to come out on top with 27% of the vote. According to analysis by broadcaster ARD, the AfD vote was highest among 35-44 year-olds, at 16%, and lowest among those above the age of 70, at 7%.
Beatrix van Storch, one of the party’s leaders, told the BBC that the party would start parliamentary debates “on migration, Islam and ever closer union” within the EU
AP