How could I split this text at certain points outside of certain things?

I have created the solution: https://repl.it/@Programit/Gettokens#main.py

Good!

But, anyways, you can split by regexp.


import re

str = '63, foo,    bar,,,,, apple'

# Split to array
arr = re.split(',+\s*', str)
print(arr)

This does not work in most cases.

of course it won't

using a split will never work for all cases

Coder100

you're going to have to rethink how you do it.

My whole language works with this.
I would have to redesign the lang

no you don't

only the lexer

but of course, you will have to eventually if you are using split based lexing.

My parser uses it too.

@Coder100

Coder100

A PARSER USES YOUR TOKENS THAT YOU HAVE GENERATED THERE IS ABSOLUTELY NO WAY IT PARSES ANY TEXT WHATSOEVER

Some people look at my code and think it doesn't even have a lexer or parser, so my code is probably very different from what you are used to.

well, you are most likely not making a real language if it has no parser

It works, and it is real. Even though with the earlier version of this it could not do things like adding the return of a function with a variable.
If I get this function then it will be fully 100% functional.

well just wanting to say that no real language uses .split lexing and no real language ever should use .split while parsing

ApoorvSingal

google "How to make a lexer in py"