Ask coding questions

← Back to all posts
How could I split this text at certain points outside of certain things?
h
hg0428

I need to make something that can do:

I need this gettokens() function.

I have not been successful in making one though, neither has anyone else so far.

Please help, I need this.

Voters
ReneganRonin
zplusfour
GameWinner6025
hg0428
Comments
hotnewtop
hg0428
PattanAhmed

@hg0428 Good!

Coder100

But, anyways, you can split by regexp.

Read more

hg0428

This does not work in most cases.
@Coder100

Coder100

of course it won't @hg0428

Coder100

@hg0428 using a split will never work for all cases

Coder100

you're going to have to rethink how you do it. @hg0428

hg0428

My whole language works with this.
I would have to redesign the lang @Coder100

Coder100

@hg0428 no you don't

Coder100

only the lexer @hg0428

Coder100

but of course, you will have to eventually if you are using split based lexing. @hg0428

hg0428

My parser uses it too. @Coder100

Coder100

A PARSER USES YOUR TOKENS THAT YOU HAVE GENERATED THERE IS ABSOLUTELY NO WAY IT PARSES ANY TEXT WHATSOEVER @hg0428

hg0428

Some people look at my code and think it doesn't even have a lexer or parser, so my code is probably very different from what you are used to.

@Coder100

RohilPatel
Coder100

well, you are most likely not making a real language if it has no parser @hg0428

hg0428

It works, and it is real. Even though with the earlier version of this it could not do things like adding the return of a function with a variable.
If I get this function then it will be fully 100% functional.
@Coder100

Coder100
Coder100

well just wanting to say that no real language uses .split lexing and no real language ever should use .split while parsing @hg0428

ApoorvSingal

google "How to make a lexer in py"

Coder100

No matter what that's never going to work, suppose you had a string that contained the special character.

RohilPatel

Ur right about that one @Coder100

hg0428

With the past version of this function it did not have that problem @Coder100

hg0428
Coder100

Why would you lex by split?

RohilPatel

That's a fair point @Coder100